TensorFlow by Google Machine Learning Foundations: Ep #8 - Tokenization for Natural Language Process
![TensorFlow by Google Machine Learning Foundations: Ep #8 - Tokenization for Natural Language Process](https://qiniu.meowparty.cn/coder.2023/2024-12-21/Lesson-6618fc247eda7.png)
机器学习基础:第 8 集 - 自然语言处理的标记化
![www.zeeklog.com - TensorFlow by Google Machine Learning Foundations: Ep #8 - Tokenization for Natural Language Process](https://qiniu.meowparty.cn/coder.2023/2024-12-21/Lesson-6618fc247eda7.png)
![www.zeeklog.com - TensorFlow by Google Machine Learning Foundations: Ep #8 - Tokenization for Natural Language Process](https://qiniu.meowparty.cn/coder.2023/2024-12-21/Lesson-15ad36dbff90c.png)
![www.zeeklog.com - TensorFlow by Google Machine Learning Foundations: Ep #8 - Tokenization for Natural Language Process](https://qiniu.meowparty.cn/coder.2023/2024-12-21/Lesson-a25aa178e7e5f.png)
![www.zeeklog.com - TensorFlow by Google Machine Learning Foundations: Ep #8 - Tokenization for Natural Language Process](https://qiniu.meowparty.cn/coder.2023/2024-12-21/Lesson-ae3bb8019a5d1.png)
![www.zeeklog.com - TensorFlow by Google Machine Learning Foundations: Ep #8 - Tokenization for Natural Language Process](https://qiniu.meowparty.cn/coder.2023/2024-12-21/Lesson-559732c6914bc.png)
![www.zeeklog.com - TensorFlow by Google Machine Learning Foundations: Ep #8 - Tokenization for Natural Language Process](https://qiniu.meowparty.cn/coder.2023/2024-12-21/Lesson-3fa3619b6019c.png)
![www.zeeklog.com - TensorFlow by Google Machine Learning Foundations: Ep #8 - Tokenization for Natural Language Process](https://qiniu.meowparty.cn/coder.2023/2024-12-21/Lesson-6936ee88e67e8.png)
![www.zeeklog.com - TensorFlow by Google Machine Learning Foundations: Ep #8 - Tokenization for Natural Language Process](https://qiniu.meowparty.cn/coder.2023/2024-12-21/Lesson-e733ee0487e3a.png)
![www.zeeklog.com - TensorFlow by Google Machine Learning Foundations: Ep #8 - Tokenization for Natural Language Process](https://qiniu.meowparty.cn/coder.2023/2024-12-21/Lesson-56b8c4bc4d9f7.png)
![www.zeeklog.com - TensorFlow by Google Machine Learning Foundations: Ep #8 - Tokenization for Natural Language Process](https://qiniu.meowparty.cn/coder.2023/2024-12-21/Lesson-a9ad3613f5678.png)
![www.zeeklog.com - TensorFlow by Google Machine Learning Foundations: Ep #8 - Tokenization for Natural Language Process](https://qiniu.meowparty.cn/coder.2023/2024-12-21/Lesson-f5fa8f522615f.png)
![www.zeeklog.com - TensorFlow by Google Machine Learning Foundations: Ep #8 - Tokenization for Natural Language Process](https://qiniu.meowparty.cn/coder.2023/2024-12-21/Lesson-0b743455dd8e5.png)
![www.zeeklog.com - TensorFlow by Google Machine Learning Foundations: Ep #8 - Tokenization for Natural Language Process](https://qiniu.meowparty.cn/coder.2023/2024-12-21/Lesson-e32e655c2b396.png)
练习
from tensorflow.keras.preprocessing.text import Tokenizer
sentences = [
'i love my dog',
'I, love my cat',
'You love my dog!',
'hello, hello, hello, hello I am in China now'
]
tokenizer = Tokenizer(num_words = 100)
tokenizer.fit_on_texts(sentences)
word_index = tokenizer.word_index
print(word_index)
![www.zeeklog.com - TensorFlow by Google Machine Learning Foundations: Ep #8 - Tokenization for Natural Language Process](https://qiniu.meowparty.cn/coder.2023/2024-12-21/Lesson-009298f1d0ad1.png)
参考
https://youtu.be/f5YJA5mQD5c