Does the script exit when complete?

If you’re running on a Linux based system and have root access you can use lsof to see what processes have with files open. (Or use the psutil Python package).

From: Python-list <python-list-bounces+gweatherby=uchc.edu@python.org> on behalf of Mike Dewhirst <miked@dewhirst.com.au>
Date: Tuesday, November 29, 2022 at 2:20 AM
To: python-list@python.org <python-list@python.org>
Subject: Python script not letting go of files
I have a script which fetches a production site directly from a
Subversion repo using svn export

It runs a bunch of commands by calling this little method ...

def trycmd(cmd, log):
retcode = -1
ret = f"Trying {cmd}"
try:
retcode = os.system(cmd)
ret = f"\n{cmd} -ok-> {retcode}"
except Exception as err:
ret = f"\n{cmd} -fail-> {err}"
log.write(remove_password(ret))
return retcode

This is the fetching script (omitting variables at the top) which
appears to be keeping a finger on files which Apache wants.

with open(fetchlog, 'a') as log:
ret = f"\n\nFetching {tag}"
log.write(ret)
cmd = f"sudo rm -Rf {site_root}"
if trycmd(cmd, log) == 0:
cmd = f"sudo svn export --force --username {usr} --password
{pw} {svn_repo} {site_root}"
if trycmd(cmd, log) == 0:
# get any new dependencies
cmd = f"sudo -H pip install -r {reqfile}"
if trycmd(cmd, log) == 0:
# run any migrations shipped from the repo
cmd = f"sudo python3 {site_root}/manage.py migrate
--noinput --settings={settings}"
if trycmd(cmd, log) == 0:
# shouldn't find anything
cmd = f"sudo python3 {site_root}/manage.py
makemigrations --noinput --settings={settings}"
if trycmd(cmd, log) == 0:
# should have been done already
cmd = f"sudo python3 {site_root}/manage.py
migrate --noinput --settings={settings}"
if trycmd(cmd, log) == 0:
# remove all static files from their Apache dir
cmd = f"sudo rm -Rf {static_root}/*"
if trycmd(cmd, log) == 0:
# copy all static files to the Apache
location
cmd = f"sudo python3
{site_root}/manage.py collectstatic --noinput --settings={settings}"
if trycmd(cmd, log) == 0:
# set all permissions
cmd = f"sudo {scripts}/perms_{host}.sh"
if trycmd(cmd, log) == 0:
cmd = "sudo service apache2
restart"
if trycmd(cmd, log) == 0:
ret = f"\nFinish {tag}\n\n"
log.write(ret)
else:
print("Apache didn't restart")
else:
print("Didn't set permissions")
else:
print("Didn't collectstaic")
else:
print("Didn't delete static files")
else:
print("Didn't migrate 2")
else:
print("Didn't makemigration")
else:
print("Didn't migrate 1")
else:
print("Didn't install requirements")
else:
print("Didn't get source")
else:
print("Didn't remove site")
exit()

The problem I'm trying to fix is that after an Apache reload Apache
seems hellbent on filling up its scoreboard and running out of
resources. The kind folk on the Apache mailing list say something is
hogging the workers.

You can see the last operation above is an Apache restart. It should be
an Apache reload. Reload however lets Apache run out of resources.

Do any of you Python folks see any blunders in the above code along the
lines of not letting go of py files or static assets?

mod_wsgi is configured with 100% defaults.
mod_mpm_event conf per advice from the Apache mailing list is ...

<IfModule mpm_event_module>
ServerLimit 32
StartServers 16
MinSpareThreads 400
MaxSpareThreads 800
ThreadLimit 64
ThreadsPerChild 50
AsyncRequestWorkerFactor 2
MaxRequestWorkers 800
MaxConnectionsPerChild 0
</IfModule>

Server Version: Apache/2.4.52 (Ubuntu 2022.04) OpenSSL/3.0.2
mod_wsgi/4.9.0 Python/3.10
Server MPM: event
Server Built: 2022-09-30T04:09:50

DigitalOcean droplet 8BG RAM and lightly loaded.

Many thanks for any hints.

Cheers

Mike


--
Signed email is an absolute defence against phishing. This email has
been signed with my private key. If you import my public key you can automatically decrypt my signature and be sure it came from me. Just
ask and I'll send it to you. Your email software can handle signing.

--- SoupGate-Win32 v1.05
* Origin: fsxNet Usenet Gateway (21:1/5)

"Weatherby,Gerard" <gweatherby@uchc.edu> writes:

Do any of you Python folks see any blunders in the above code along the
lines of not letting go of py files or static assets?

No, sorry. But I had to refactor it.

*Warning* The following code is not intended to be executed,
it is just intended to be read by humans. It was not tested
and, therefore, it probably contains errors. If executed, it
may cause destruction such as deletions of files etc. Stefan
Ram (I) does not claim any copyright for his modifications.

def attempt( task, command ):
if trycmd( command, log ):
print( f'task "{task}" failed.' )
return False # failure
else:
return True # success

def main():
with open( fetchlog, 'a' )as log_:
global log
log = log_
remove_site()
del log

def remove_site():
log.write( f"\n\nFetching {tag}" )
if attempt( "remove site", f"sudo rm -Rf {site_root}", log ):
get_source()

def get_source():
if attempt( "get source", f"sudo svn export --force --username {usr} --password {pw} {svn_repo} {site_root}" ):
install_requirements()

def install_requirements():
if attempt( "install requirements", f"sudo -H pip install -r {reqfile}" ):
repo_migrations()

def repo_migrations():
if attempt( "run any migrations shipped from the repo", f"sudo python3 {site_root}/manage.py migrate --noinput --settings={settings}" ):
makemigrations()

def makemigrations():
# shouldn't find anything
if attempt( "makemigrations", f"sudo python3 {site_root}/manage.py makemigrations --noinput --settings={settings}" ):
migrate()

def migrate():
# should already be done
if attempt( "migrate 2", f"sudo python3 {site_root}/manage.py migrate --noinput --settings={settings}" ):
remove_static()

def remove_static():
if attempt( "remove all static files from their Apache dir", f"sudo rm -Rf {static_root}/*" ):
copy_static()

def copy_static():
if attempt( "copy all static files to the Apache location", f"sudo python3 {site_root}/manage.py collectstatic --noinput --settings={settings}" ):
set_permissions()

def set_permissions():
if attempt( "set all permissions", f"sudo {scripts}/perms_{host}.sh" ):
service_restart()

def service_restart():
if attempt( "restart of Apache service", f"sudo service apache2 restart" ):
log_success()

def log_success():
log.write( f"\nFinish {tag}\n\n" )

# main()
# sys.exit()

--- SoupGate-Win32 v1.05
* Origin: fsxNet Usenet Gateway (21:1/5)

"Weatherby,Gerard" <gweatherby@uchc.edu> writes:

Do any of you Python folks see any blunders in the above code along the
lines of not letting go of py files or static assets?

Er, no, I just replied to the original poster.

--- SoupGate-Win32 v1.05
* Origin: fsxNet Usenet Gateway (21:1/5)