Introduction to tensor

Definition

In TensorFlow, a tensor is a fundamental data structure representing multi-dimensional arrays or vectors. It's similar to NumPy arrays but designed specifically for deep learning computations. Tensors are the basic building blocks of TensorFlow computations.

Key concept related to tensors in TensorFlow:

Rank: The rank of a tensor refers to the number of dimensions it has. A rank-0 tensor is a scalar (a single value), a rank-1 tensor is a vector (one-dimensional array), a rank-2 tensor is a matrix (two-dimensional array), and so on.

Shape: The shape of a tensor defines the number of elements in each dimension. For example, a tensor with shape [3, 4] represents a matrix with 3 rows and 4 columns.

Data Types: TensorFlow tensors can have different data types such as float32, int32, or string, among others. The data type of a tensor determines the type of data it can hold.

Operations: TensorFlow provides a wide range of operations (ops) for manipulating tensors, such as addition, multiplication, reshaping, and more. These operations can be performed on tensors to perform various computations in neural networks.

import numpy as np
import tensorflow as tf

Basic Tensors

rank-0 or scalar tensor. The scalar tensor have no axes, it represent single value.

tensor_0 = tf.constant(4) # default dtype is int32

print(tensor_0) 

A one dimensional list of values is called as rank-1 tesnor or simply called as vector.

tensor_1 = tf.constant([2.0,3.0,4.0,5.0])
print(tensor_1)

A tensor with 2-axes is called as matrix or rank-2 tensors.

tensor_2 = tf.constant([
    [1,2],
    [3,4],
    [5,6]
])
print(tensor_2)

shape of all these tensors:

print(tensor_0.shape)
print(tensor_1.shape)
print(tensor_2.shape)

A tensor can have more than 2-axes

# A tensor can have arbitary number of axes also called as dimension of tensor.
tensor_3 = tf.constant([
    [[1,2],[3,4]],
    [[5,6],[7,8]],
    [[9,10],[11,12]]
])

print(tensor_3)

print(tensor_3.shape)

we can convert tensor into numpy array using following method:

print(np.array(tensor_2)) # using np.array()

# or we can use tensor.numpy()
x = tensor_2.numpy()
print(x)
print(type(x))

Basic Math operations on tensor

Addition

x = tf.constant([
    [1,2],
    [3,4]
    ])
y = tf.constant([
    [10,11],
    [12,13]
])

a = tf.add(x,y)
print(a)

element-wise multiplication

b = tf.multiply(x,y)
print(b)

Matrix Multiplication

c = tf.matmul(x,y)
print(c)

The above operations can be done in following ways too:

print('Addition')
print(x+y)
print('Multiplication')
print(x*y)
print('Matrix Multiplication')
print(x@y)

We can perform various types of other operations i.e ops on tensors

a = tf.constant([[5.0,6.0],[8.0,10.5]])

find largest value in given matrix

print(tf.reduce_max(a))

find the index of largest value

print(tf.math.argmax(a))

compute the softmax of given matrix

sft = tf.nn.softmax(a)
print(sft)

About tensor shapes:

Tensors have shapes. Some vocabulary:

Shape: The length (number of elements) of each of the axes of a tensor.

Rank: Number of tensor axes. A scalar has rank 0, a vector has rank 1, a matrix is rank 2.

Axis or Dimension: A particular dimension of a tensor.

Size: The total number of items in the tensor, the product of the shape vector's elements.

Understanding higher dimensional tensor

tensor_4 = tf.zeros([3,2,4,5])

print(tensor_4)

print("Type of every element:", tensor_4.dtype)
print("Number of axes:", tensor_4.ndim)
print("Shape of tensor:", tensor_4.shape)
print("Elements along axis 0 of tensor:", tensor_4.shape[0])
print("Elements along the last axis of tensor:", tensor_4.shape[-1])
print("Total number of elements (3*2*4*5): ", tf.size(tensor_4).numpy())

Generally, in deep learning models a higher dimensional tensor as below, can be understood as:

x = tf.zeros([3 ,2, 4, :5])

here,

3 => Batch

2 => Height

4 => width

5 => number of features 

Indexing

Single-axis indexing

TensorFlow follows standard Python indexing rules, similar to indexing a list or a string in Python, and the basic rules for NumPy indexing.

indexes start at 0

negative indices count backwards from the end

colons, :, are used for slices: start:stop:step

tensor_1 = tf.constant([1,2,3,4,5,6,7,8,9,10])

print(tensor_1.numpy())

print(tensor_1[1].numpy())
print(tensor_1[-1].numpy())
print(tensor_1[5].numpy())

# slicing operations
print(tensor_1[2:7].numpy())
print(tensor_1[5:].numpy())

Multi-axis indexing

indexing multi-axis is sames as single axis, whereas in multi-axis indexing rules applies independently to each of its axis

a = tf.constant([[1.0,2.0,3.0],[4.0,5.0,6.0]])
print(a.numpy())

print(a[1,1].numpy())
print(a[1:,1:].numpy())
print(a[0,:].numpy())
print(a[:,0].numpy())

Manipulating shapes of tensor

x = tf.constant([[10],[20],[30]])

print(x)
print(x.shape)
y = tf.reshape(x,(1,3))
print(y.numpy())
print(y.shape)

reshaping rank-3 tensor

x = tf.constant([[[1,2,3,4],[5,6,7,8]],[[10,20,30,40],[50,60,70,80]]])
print(x)

print(x.shape)

let us flatten the tensor:

y = tf.reshape(x,[-1])
print(y.numpy())

z = tf.reshape(x,[2,-1])
print(z)

print(tf.reshape(x,[2*2,-1]).numpy())

Note: Reshaping will "work" for any new shape with the same total number of elements, but it will not do anything useful if you do not respect the order of the axes.

try:
  tf.reshape(x,[7,-1])
except Exception as e:
  print(e)

Tensor DType

when creating tensor from python object, optionally specify the data type

a = tf.constant([10,20,30]) # default int32
print(a.dtype)

b = tf.constant([1.0,2.0,3.0])
print(b.dtype)

# specify dtype as argument
x = tf.constant([10,20,30],dtype=tf.int64)
print(x)
print(x.dtype)

y = tf.constant([5.56,7.89,3.20,4.0],dtype=tf.float16)
print(y)

typecasting tensor

a = tf.cast(y,dtype=tf.int16) # results in data loss
print(a)

Broadcasting

Broadcasting is a concept borrowed from the equivalent feature in NumPy. In short, under certain conditions, smaller tensors are "stretched" automatically to fit larger tensors when running combined operations on them.

a = tf.constant([4,5,6])
b = tf.constant([3])
c = tf.constant([8,9,10])

print(tf.multiply(a,b))
print(tf.add(a,b))

The simplest and most common case is when you attempt to multiply or add a tensor to a scalar. In that case, the scalar is broadcast to be the same shape as the other argument.

Ragged Tensors

A tensor with variable numbers of elements along some axis is called "ragged". Use tf.ragged.RaggedTensor for ragged data.

ragged_l = [
    [10,20,30],
    [40],
    [50,60]
]

try:
  r = tf.constant(ragged_l)
except Exception as e:
  print(e)

Can't convert non-rectangular Python sequence to Tensor.

#instead we can use ragged tensor as follows:
try:
  r = tf.ragged.constant(ragged_l)
  print(r)
except Exception as e:
  print(e)

# The shape of a tf.RaggedTensor will contain some axes with unknown lengths:

print(r.shape)

String tensors

tf.string is a dtype, which is to say you can represent data as strings (variable-length byte arrays) in tensors.The strings are atomic and cannot be indexed the way Python strings are. The length of the string is not one of the axes of the tensor.

string_tensor_0 = tf.constant('hello')
print(string_tensor_0)

# here strings are byte strings
string_tensor_1 = tf.constant(['hello','world','Thank You','welcome'])
print(string_tensor_1) 

Basic string operations

split strings

s_t = tf.constant('hello how are you?')
splitted = tf.strings.split(s_t,sep=" ")
print(splitted)
print(splitted.numpy())

x = tf.constant([['ab cd'],["efg hijkl hikl"]]) # results in ragged tensor
splitted_x = tf.strings.split(x,sep=" ")
print(splitted_x)

string to number

a = tf.constant("1 100 120 567")
b = tf.strings.to_number(tf.strings.split(a,sep=" "))
print(b)

Although you can't use tf.cast to turn a string tensor into numbers, you can convert it into bytes, and then into numbers.

byte_strings = tf.strings.bytes_split(tf.constant("apple"))
print(byte_strings)

byte_ints = tf.io.decode_raw(tf.constant("apple"), tf.uint8)
print("Bytes:", byte_ints)