Htykuut

I am self-teaching myself about tf.data API. I am using MNIST dataset for binary classification. The training x and y data is zipped together in the full train_dataset. Chained along together with this zip method is first the batch() dataset method. the data is batched with a batch size of 30. Since my training set size is 11623, with batch size 128, I will have 91 batches. The size of the last batch will be 103 which is fine since this is LSTM. Additionally, I am using drop-out. When I compute batch accuracy, I am turning off the drop-out.

The full code is given below:

#Ignore the warnings

import warnings

warnings.filterwarnings("ignore")



import pandas as pd

import tensorflow as tf

import numpy as np



import matplotlib.pyplot as plt

plt.rcParams['figure.figsize'] = (8,7)



from tensorflow.examples.tutorials.mnist import input_data

mnist = input_data.read_data_sets("MNIST_data/")



Xtrain = mnist.train.images[mnist.train.labels < 2]

ytrain = mnist.train.labels[mnist.train.labels < 2]



print(Xtrain.shape)

print(ytrain.shape)



#Data parameters

num_inputs = 28

num_classes = 2

num_steps=28



# create the training dataset

Xtrain = tf.data.Dataset.from_tensor_slices(Xtrain).map(lambda x: tf.reshape(x,(num_steps, num_inputs)))

# apply a one-hot transformation to each label for use in the neural network

ytrain = tf.data.Dataset.from_tensor_slices(ytrain).map(lambda z: tf.one_hot(z, num_classes))

# zip the x and y training data together and batch and Prefetch data for faster consumption

train_dataset = tf.data.Dataset.zip((Xtrain, ytrain)).batch(128).prefetch(128)



iterator = tf.data.Iterator.from_structure(train_dataset.output_types,train_dataset.output_shapes)

X, y = iterator.get_next()



training_init_op = iterator.make_initializer(train_dataset)





#### model is here ####



#Network parameters

num_epochs = 2

batch_size = 128

output_keep_var = 0.5



with tf.Session() as sess:

    init.run()



    print("Initialized")

    # Training cycle

    for epoch in range(0, num_epochs):

        num_batch = 0

        print ("Epoch: ", epoch)

        avg_cost = 0.

        avg_accuracy =0

        total_batch = int(11623 / batch_size + 1)

        sess.run(training_init_op)

       while True:

            try:

                _, miniBatchCost = sess.run([trainer, loss], feed_dict={output_keep_prob: output_keep_var})

                miniBatchAccuracy = sess.run(accuracy, feed_dict={output_keep_prob: 1.0})

               print('Batch %d: loss = %.2f, acc = %.2f' % (num_batch, miniBatchCost, miniBatchAccuracy * 100))

                num_batch +=1

            except tf.errors.OutOfRangeError:

                break

When I run this code, it seems it is working and printing:

Batch 0: loss = 0.67276, acc = 0.94531

Batch 1: loss = 0.65672, acc = 0.92969

Batch 2: loss = 0.65927, acc = 0.89062

Batch 3: loss = 0.63996, acc = 0.99219

Batch 4: loss = 0.63693, acc = 0.99219

Batch 5: loss = 0.62714, acc = 0.9765

......

......

Batch 39: loss = 0.16812, acc = 0.98438

Batch 40: loss = 0.10677, acc = 0.96875

Batch 41: loss = 0.11704, acc = 0.99219

Batch 42: loss = 0.10592, acc = 0.98438

Batch 43: loss = 0.09682, acc = 0.97656

Batch 44: loss = 0.16449, acc = 1.00000

However, as one can see easily, there is something wrong. Only 45 batches are printed not 91 and I do not know why this is happening. I tried so many things and I think I am missing something out.

I can use repeat() function but I do not want that because I have redundant observations for last batches and I want LSTM to handle it.

This is an annoying pitfall when defining a model based directly on the get_next() output of a tf.data iterator. In your loop, you have two sess.run calls, both of which will advance the iterator by one step. This means each loop iteration actually consumes two batches (and also your loss and accuracy calculations are computed on different batches).

Not entirely sure if there is a "canonical" way of fixing this, but you could

• compute the accuracy in the same run call as the cost/training step. This would mean that the accuracy calculation is also affected by the dropout mask, but since it's an approximate value based on only one batch, that shouldn't be a huge issue.


• define your model based on a placeholder instead, and in each loop iteration run the get_next op itself, then feed the resulting numpy arrays (i.e. the batch) into the loss/accuracy computations.