Selenium and asyncio

Question

What is the best way to go about making this work? Since Selenium doesn't work well with asyncio, the bot will eventually die. It works as intended, but will crash eventually. I do understand what the issue is, especially since selenium is so heavy, but is there a good work around?

I get the error message: Task was destroyed but it is pending!

I don't have anything else, no traceback. I have to reboot the bot (discord) and it'll work again for a bit.

@bot.command(pass_context=True)
async def weather(ctx, arg):
    if ctx:
        list = ('lf','oahu')
        await bot.say('*Loading webdriver.....*')
        if arg in list:
            if arg == 'lf':
                website = '--'
                name = 'Los Feliz Map'
            if arg == 'oahu':
                website = '--'
                name = 'Oahu Map'
            load_site = webdriver.Chrome()
            # load site on chrome
            load_site.get(website)
            await bot.say('*Loading ' + str(name) + '...*')
            load_site.find_element_by_xpath('//*[@id="main-map"]/div[2]/div[1]/div/a[2]').click()
            load_site.find_element_by_xpath('//*[@id="main-map"]/div[2]/div[2]').click()
            load_site.find_element_by_xpath('//*[@id="main-map"]/div[2]/div[2]/div/form/div[3]/label[6]/div/span').click()
            await asyncio.sleep(2) #sleep to ensure weather loads on site before screenshot
            load_site.save_screenshot('weather.png')
            await bot.say('*Taking screenshot and saving image....*')
            await bot.send_file(ctx.message.channel, 'weather.png')
            load_site.quit()
            print('Succesfully sent image of ' + str(arg) + ' - ' + str(ctx.message.author.name))

I left out the website because it's a private one.

I don't think you're going to have much success. Blocking APIs like selenium really don't work in asynchronous applications. Your best bet would be to use a purpose built asynchronous library. arsenic seems to be roughly that, but you'd have to experiment with it. — Patrick Haugh
Yeah I found arsenic, definitely gonna have to do some reading on that. Luckily "click" and "screenshot" still work with arsenic, at a quick glance I didn't see anything about searching with xpath though. will look again later tonight, thank you. — sb2894
You can use run_in_executor to run blocking code in it's own thread without blocking. You can then run all your code in the thread and return what is needed. See example here: stackoverflow.com/questions/53587063/… — Benjin
You could try using playwright, which supports asynchronous usage. — Jacob Lee
Using a browser is a bad idea (in general), your better off using aiohttp and beautiful soup to webscrape the website — mcdonalds291

mcdonalds291 mcdonalds291 · Accepted Answer · 2021-06-24T17:26:45

Short answer: its not possible. If you somehow manage to get this to work, it wont work well.

Long answer: Selenium is a blocking API, you can't really use it with asynchronous scripts. If you really want to use a web browser you can try to find an alternative like https://pypi.org/project/aioselenium/.

Selenium and asyncio

2 Answers