Tokens in Python 3.9 / Garg's Academy

Tokens: A token is the smallest individual unit in a python program. There are 5 types of tokens in Python:

keywords
Identifier
Literal/Constant
Operator
Punctuator/Separator

Keywords: These are reserved words in python used for some specific purpose. They cant be used as variable names, function names, or any other general purpose. There are 36 keywords in Python 3.9

How to check List of Keywords in Python 3.9

Method 1

>>>import keyword

>>> len(keyword.kwlist)
36

>>> keyword.kwlist
['False', 'None', 'True', '__peg_parser__', 'and', 'as', 'assert', 'async', 'await', 'break', 'class', 'continue', 'def', 'del', 'elif', 'else', 'except', 'finally', 'for', 'from', 'global', 'if', 'import', 'in', 'is', 'lambda', 'nonlocal', 'not', 'or', 'pass', 'raise', 'return', 'try', 'while', 'with', 'yield']

Method 2

>>>help("keywords")
Here is a list of the Python keywords.  Enter any keyword to get more help.

False               break               for                 not
None                class               from                or
True                continue            global              pass
__peg_parser__      def                 if                  raise
and                 del                 import              return
as                  elif                in                  try
assert              else                is                  while
async               except              lambda              with
await               finally             nonlocal            yield

Classification of Keywords

Value Keywords: (3) True, False, None
Operator Keywords: (5) and, or, not, in, is
Control Flow Keywords: (3) if, elif, else
Iteration Keywords: (5) for, while, break, continue, else
Structure Keywords: (6) def, class, with, as, pass, lambda
Returning Keywords: (2) return, yield
Import Keywords: (3) import, from, as
Exception-Handling Keywords: (6) try, except, raise, finally, else, assert
Asynchronous Programming Keywords: (2) async, await
Variable Handling Keywords: (3) del, global, nonlocal
Other keywords: (1) __peg_parser__

Note:
Two keywords as and else are used for different purposes.
else is used in decision making, looping and exception handling.
as is used as structure keyword as well as for import statements.
All keywords are written in small case letters except None, True and False

iskeyword() : It is a function of keyword module and used to check whether a string is a keyword or not in python.

>>> import keyword
>>> keyword.iskeyword("for")
True
>>> keyword.iskeyword("For")
False

Identifier: Identifiers are the names given to program elements like any variable, function, class, list, methods, etc. for their identification. Python is a case-sensitive language and treats uppercase and lowercase letters differently. Rules for naming an identifier:-
1. It is any combination of alphabets, digits and _(underscore)
2. No other symbol is allowed.
3. It must not start with a digit
4. It must not be a keyword
5. It is case sensitive

Examples of valid identifiers:
roll_no
rollno
rollno1
For (for is keyword but not For)
true (True is keyword but not true)

Examples of invalid identifiers:
roll no (Space not allowed)
rollno. (. not allowed)
1rollno (must not start with a digit)
for (must not be a keyword)
True (must not be keyword)

isidentifier() : It is a function of strings and used to check whether a string is a valid identifier or not in python.

>>> "1roolno".isidentifier()
False

>>> "rollno1".isidentifier()
True

>>> "roll no".isidentifier()
False

>>> "For".isidentifier()
True

>>># Notice that it will return True even if it is a keyword 
>>># as it does not check if the string is a keyword
>>> "for".isidentifier()
True

Literals/ Constants: These are fixed values that don’t change during program execution. Following types of literals are supported in python.
1. Numeric (Integer, Float, Complex)
2. Boolean
3. Sequences/ Collections (List, Tuple, Set, Strings)
4. Mapping ( Dictionary)
5. Special Literal None

1. Numeric Literal: These are literals in the form of numbers. Python supports Integer, Float and Complex Literals.

a) Integer Literal: It is any combination of digits along with +/- sign. It includes both positive and negative numbers along with 0. It doesn’t include fractional parts.

>>> a=4398
>>> a
4398
>>> a=-34534
>>> a
-34534
>>> a=0
>>> a
0

b) Float Literal: It includes both positive and negative real numbers. It also includes fractional parts. There are two forms to represent float literals: I) standard form and II) mantissa exponent form.

I) Standard Form: Any number with or without +/- sign having decimal point

>>> a=54.645
>>> a
54.645
>>> a=-324.654
>>> a
-324.654
>>> a=9.
>>> a
9.0
>>> a=.7
>>> a
0.7

II) Mantissa Exponent Form: mantissa may have any number with or without +/- sign having decimal point, and exponent must be a positive or negative integer followed by e/E sign. Internally python stores the number in normalized form and using e sign for exponent.

>>> a=44.3432e34
>>> a
4.43432e+35
>>> a=0.00324E-45
>>> a
3.24e-48
>>> a=0.00324E-4.5
SyntaxError: invalid syntax

There must not be any space between mantissa, e and exponent part.

>>> a=44.3432 e 34
SyntaxError: invalid syntax
>>> a=44.3432 e34
SyntaxError: invalid syntax

c) Complex Literal: It is of the form a + bj , here a represents the real part and b represents the complex part. Both j and J are allowed but internally Python stores j.

>>> a=8+4j
>>> a=8+4J
>>> a
(8+4j)

There must not be any space between b and j

>>> a=9+5 j
SyntaxError: invalid syntax

2. Boolean Literals: There are two Literals True and False which are known as Boolean Literals.

>>> a=True
>>> a
True

>>> b=False
>>> b
False

3. Literal collections: Python provides 4 different types of literal collections:

List literals : Comma separated list of items (may be of same / different type) enclosed in square brackets [ and ]
Example: [“gargs” ,”academy”, 9467863365]
Tuple literals: Comma separated list of items (may be of same / different type) enclosed in paranthesis ( and )
Example: (“gargs” ,”academy”, 9467863365)
Set literals : Comma separated list of unordered and unique items (may be of same / different type) enclosed in braces { and }
Example: {“gargs” ,”academy”, 9467863365}
String Literal: Any combination of characters or text written in single, double, or triple quotes. Internally strings are stored in single quotes.

>>> list1=["gargs" ,"academy", 9467863365]
>>> list1
['gargs', 'academy', 9467863365]

>>> tuple1=("gargs" ,"academy", 9467863365)
>>> tuple1
('gargs', 'academy', 9467863365)

>>> set1={"gargs" ,"academy", 9467863365}
>>> set1
{'academy', 9467863365, 'gargs'}

>>> str='gargs academy'
>>> str
'gargs academy'

>>> str="gargs academy"
>>> str
'gargs academy'

Triple quotes ”’ and “”” are used for multi line strings.

>>> str="""gargs
academy"""
>>> str
'gargs\nacademy'

>>> str='''gargs
academy'''
>>> str
'gargs\nacademy'

For multi line strings, statement continuation mark \ is used. If we want space in between the strings, we must give space explicitly.

>>> str="""gargs\
academy"""
>>> str
'gargsacademy'

>>> str="""gargs\
 academy"""
>>> str
'gargs academy'

It can also include binary, octal, hexadecimal string literal.

>>> a=bin(10)
>>> a
'0b1010'
>>> a=oct(10)
>>> a
'0o12'
>>> a=hex(10)
>>> a
'0xa'

4. Dictionary literals It is a comma separated list of key : value paired form of unordered items (may be of same / different type) enclosed in braces { and }
Example: {“gargs” : 1 , 2 : “academy”, “phone” : 9467863365}

>>> dict1= {"gargs" : 1 , 2 : "academy", "phone" : 9467863365}
>>> dict1
{'gargs': 1, 2: 'academy', 'phone': 9467863365}

5. Special Literal: None is used to define a null or missing value

>>> a=None
>>> a
>>> print(a)
None

Operator: It is a symbol(s) that performs some action. Python supports following types of operators:

Arithmetic operators : + , – , * , / , // , % , **
Assignment operators: = , += , -= , *= , /= , %= , //= , **= , &= , |= , ^= , >>= , <<=
Comparison/Relational operators: < , <= , > , >= , = , !=
Logical operators: and , or , not
Identity operators: is , is not
Membership operators: in , not in
Bitwise operators: &, | ,~ , ^ , >> , <<

Punctuators/ Separators: It is used to separate different parts of the program.
Example: ! ” # $ \ ‘ ( ) , . : ; ? @ [ \ ] _` { }

Related Posts