PyTorch简单笔记

张量(tensor)基础

构造张量

• 可以从list构造，用[]访问和修改，下标访问直接得到的是PyTorch Scalar
• 有.dim(), .shape方法，.item()可以将PyTorch scalar转换为普通的python scalar

# Create a two-dimensional tensor
b = torch.tensor([[1, 2, 3], [4, 5, 5]])
print('Here is b:')
print(b)
print('rank of b:', b.dim())
print('b.shape: ', b.shape)

# Access elements from a multidimensional tensor
print()
print('b[0, 1]:', b[0, 1])
print('b[1, 2]:', b[1, 2])

# Mutate elements of a multidimensional tensor
b[1, 1] = 100
print()
print('b after mutating:')
print(b)

内置的构造

• torch.zeros: Creates a tensor of all zeros
• torch.ones: Creates a tensor of all ones
• torch.rand: Creates a tensor with uniform random numbers
• torch.eye: 创建一个单位张量
• torch.empty: 使用无用数据进行填充
• torch.full: 所有元素相同的张量
• 以及其他的选择 in the documentation

数据类型

• large set of numeric datatypes
• tensor的.dtype属性可以查看类型，也可以指定类型
• 也可以使用.to(), .float()之类的方法进行类型转换

# Force a particular datatype
x0 = torch.tensor([1, 2], dtype=torch.float32)  # 32-bit float
y0 = torch.eye(3, dtype=torch.int64)

x1 = x0.float()  # Cast to 32-bit float
x2 = x0.double() # Cast to 64-bit float
x3 = x0.to(torch.float32) # Alternate way to cast to 32-bit float
x4 = x0.to(torch.float64) # Alternate way to cast to 64-bit float

• 构造一些与已有tensor相同类型的tensor

x0 = torch.eye(3, dtype=torch.float64)  # Shape (3, 3), dtype torch.float64
x1 = torch.zeros_like(x0)               # Shape (3, 3), dtype torch.float64
x2 = x0.new_zeros(4, 5)                 # Shape (4, 5), dtype torch.float64
x3 = torch.ones(6, 7).to(x0)            # Shape (6, 7), dtype torch.float64)

tensor索引

切片索引

• tensor支持[start:stop:step]式的索引
• 切片得到的是原tensor的一个view，改变切片也会改变原tensor，规避方法是采用.clone()

# Create the following rank 2 tensor with shape (3, 4)
a = torch.tensor([[1,2,3,4], [5,6,7,8], [9,10,11,12]])
print('Original tensor')
print(a)

row_r1 = a[1, :]    # Rank 1 view of the second row of a  
row_r2 = a[1:2, :]  # Rank 2 view of the second row of a

似乎采用a[[1], :]可以得到一个新的tensor而非原tensor的view，同时形式和a[1:2, :]类似

• 支持a[1:3, 2: 4] = 2形式的赋值

整型索引

• 使用索引数组得到全新的tensor

  • a[idx0, idx1]相当于

    torch.tensor([
      a[idx0[0], idx1[0]],
      a[idx0[1], idx1[1]],
      ...,
      a[idx0[N - 1], idx1[N - 1]]
    ])

布尔tensor索引

• 使用一个torch.bool类型的mask进行索引

a = torch.tensor([[1,2], [3, 4], [5, 6]])
print('Original tensor:')
print(a)

# Find the elements of a that are bigger than 3. The mask has the same shape as
# a, where each element of mask tells whether the corresponding element of a
# is greater than three.
mask = (a > 3)
print('\nMask tensor:')
print(mask)

# We can use the mask to construct a rank-1 tensor containing the elements of a
# that are selected by the mask
print('\nSelecting elements with the mask:')
print(a[mask])
# tensor([4, 5, 6])

# We can also use boolean masks to modify tensors; for example this sets all
# elements <= 3 to zero:
a[a <= 3] = 0
print('\nAfter modifying with a mask:')
print(a)
# tensor([[0, 0],
#         [0, 4],
#         [5, 6]])

改变形状

View

• .view()可以将tensor重塑形状，并返回该tensor的一个view

  • .view(-1)会令-1的维度大小保持和原来相同

  似乎是按行展开重塑的

交换维度

• .t()直接进行转置
• .transpose()选择任意两个维度进行交换
• .permute()重新将维度进行排序

# Create a tensor of shape (2, 3, 4)
x0 = torch.tensor([
     [[1,  2,  3,  4],
      [5,  6,  7,  8],
      [9, 10, 11, 12]],
     [[13, 14, 15, 16],
      [17, 18, 19, 20],
      [21, 22, 23, 24]]])

# Swap axes 1 and 2; shape is (2, 4, 3)
x1 = x0.transpose(1, 2)

# Permute axes; the argument (1, 2, 0) means:
# - Make the old dimension 1 appear at dimension 0;
# - Make the old dimension 2 appear at dimension 1;
# - Make the old dimension 0 appear at dimension 2
# This results in a tensor of shape (3, 4, 2)
x2 = x0.permute(1, 2, 0)

Contiguous

• 一串改变tensor形状的操作连在一起可能会导致奇妙的错误，这是由于内部实现导致的
  • 可以参见this blog post by Edward Yang
• 可以通过.reshape()代替.view()或者在.view()之前添加.contiguous()来规避问题

x0 = torch.randn(2, 3, 4)

try:
  # This sequence of reshape operations will crash
  x1 = x0.transpose(1, 2).view(8, 3)
except RuntimeError as e:
  print(type(e), e)
  
# We can solve the problem using either .contiguous() or .reshape()
x1 = x0.transpose(1, 2).contiguous().view(8, 3)
x2 = x0.transpose(1, 2).reshape(8, 3)
print('x1 shape: ', x1.shape)
print('x2 shape: ', x2.shape)

张量运算

逐元素运算

• 有重载的+ - * / **，并且也有对应的方法
• .sqrt()，.sin()之类的方法，可以参考in the documentation

Reduction operation

• 有.sum()，.min()，.max()，.mean()，之类的函数，完整的列表in the documentation

x = torch.tensor([[1, 2, 3], 
                  [4, 5, 6]], dtype=torch.float32)

print('\nSum over entire tensor:')
print(torch.sum(x))
print(x.sum())

# We can sum over each row:
print(torch.sum(x, dim=0))
print(x.sum(dim=0))

# Sum over each column:
print(torch.sum(x, dim=1))
print(x.sum(dim=1))

#Sum of each row:
#tensor([5., 7., 9.])
#tensor([5., 7., 9.])

#Sum of each column:
#tensor([ 6., 15.])
#tensor([ 6., 15.])

• dim参数描述要沿着哪个维度做缩减，也就是说做完缩减这个维度会删除
• 如果keepdim=True，就会仍然保留这个维度，且该维度的值为1

# Create a tensor of shape (128, 10, 3, 64, 64)
x = torch.randn(128, 10, 3, 64, 64)

# Take the mean over dimension 1; shape is now (128, 3, 64, 64)
x = x.mean(dim=1)
# Take the sum over dimension 2; shape is now (128, 3, 64)
x = x.sum(dim=2)

# Take the mean over dimension 1, but keep the dimension from being eliminated
# by passing keepdim=True; shape is now (128, 1, 64)
x = x.mean(dim=1, keepdim=True)

矩阵运算

• torch.dot: 向量内积
• torch.mm: 矩阵乘矩阵
• torch.mv: 矩阵乘向量
• torch.addmm / torch.addmv: 矩阵乘矩阵/向量后再加一个偏置(bias)
• torch.bmm / torch.baddmm: 三维张量乘法
• torch.matmul: 通用的张量乘法，表现像numpy的np.dot
• 完整的函数列表in the documentation

Vectorization

• 尽量避免显式地使用循环，而是调用Pytorch的运算来隐式完成循环操作，这通常会快得多，这种代码风格称为vectorization

广播

• 广播的规则如下

  • 把低维度的张量扩展成高维度的，新增的维度大小为1
  • 如果两个张量在某维度上大小相同或者某个张量在该维度大小为1，称它们在这个维度上是可比较的
  • 当两个张量在所有维度都可比较时，可以进行广播
  • 广播之后，两个张量每个维度的大小都是最大的那个
  • 如果一个张量在某维度大小是1，另外一个大于1，广播的表现则是将前者沿着该维度进行复制
  • 详尽解释在documentation

  第一步中的添加维度似乎是从头（第0维）插入进去

• 广播通常隐式进行，但也有显式的方法torch.broadcast_tensors

Out-of-place vs in-place

• 大多数PyTorch运算被分成两类

  • Out-of-place operators: 返回一个新的tensor，大多数PyTorch运算表现如此

  • In-place operators: 修改并返回输入的tensor。带下划线的实例方法 （比如add_()）是in-place的。在 torch 命名空间内的运算可以用 out= 参数使其in-place

• 通常应该避免使用in-place因为他们在自动求导计算梯度的时候可能会出问题

GPU加速

• tensor有device属性用来表征tensor储存在哪里，CPU还是CUDA(for Nvidia GPUs)
• .to()方法可以转换tensor的device属性，也可以用.cuda()和.cpu()，更加方便