How To Use Different UserAgent In Selenium

Hi Guys,
Using the same user agent consistently in Selenium for web scraping can result in bot detection, blocking, access restrictions, and more. In this lesson, we’ll explore how to dynamically rotate user agents every time Selenium opens the browser.

Why is changing user agent useful?

Avoids Bot Detection
Overcome Access Restrictions

Lets Start

Rather than maintaining an extensive list of user agents for rotation, which could be quite labor-intensive, we can streamline the process using a package called ‘fake-useragent.’ This package offers a diverse collection of user agents, making it more efficient and practical for us to switch between them dynamically.

pip install fake-useragent

after installing. lets look at what you can do with this package. create a file called fake_useragent_test.py

# lets first import it
from fake_useragent import UserAgent

# to randomize everything: the operating system, browser type and other all you have to do is
ua = UserAgent()
user_agent = ua.random

# if you want to specify your own list of browsers which the user agent should be from
ua = UserAgent(browsers=['firefox', 'chrome','safari'])
user_agent = ua.random

# if you want to specify your own list of operating systems you can use this
ua = UserAgent(os=['windows', 'macos','linux'])
user_agent = ua.random

Now that we have a some understanding on how to use this package lets now add it to our selenium. Do not forget to install selenium

from selenium import webdriver
from selenium.webdriver.chrome.options import Options
from selenium.webdriver.chrome.service import Service
from webdriver_manager.chrome import ChromeDriverManager
from webdriver_manager.core.os_manager import ChromeType
from fake_useragent import UserAgent

ua = UserAgent()
user_agent = ua.random

options = Options()
options.add_argument("--disable-dev-shm-usage")
options.add_argument("--disable-infobars")
options.add_argument("--enable-file-cookies")
options.add_argument(f'--user-agent={user_agent}')

driver = webdriver.Chrome(   service=Service(ChromeDriverManager(chrome_type=ChromeType.CHROMIUM).install()), options=options
        )

driver.get("https://www.example.com")