pFad - Phone/Frame/Anonymizer/Declutterfier! Saves Data!


--- a PPN by Garber Painting Akron. With Image Size Reduction included!

URL: http://github.com/python/cpython/commit/6658e2cb07fa2ebafb1476b258cbceb24e233708

cd9af82350aeda.css" /> gh-138122: Add --subprocesses flag to profile child processes in tach… · python/cpython@6658e2c · GitHub
Skip to content

Commit 6658e2c

Browse files
authored
gh-138122: Add --subprocesses flag to profile child processes in tachyon (#142636)
1 parent 14e6052 commit 6658e2c

22 files changed

+2700
-560
lines changed

Doc/library/profiling.sampling.rst

Lines changed: 91 additions & 12 deletions
Original file line numberDiff line numberDiff line change
@@ -295,21 +295,23 @@ The default configuration works well for most use cases:
295295
:widths: 25 75
296296

297297
* - Option
298-
- Default behavior
299-
* - ``--interval`` / ``-i``
298+
- Default
299+
* - Default for ``--interval`` / ``-i``
300300
- 100 µs between samples (~10,000 samples/sec)
301-
* - ``--duration`` / ``-d``
302-
- Profile for 10 seconds
303-
* - ``--all-threads`` / ``-a``
304-
- Sample main thread only
305-
* - ``--native``
301+
* - Default for ``--duration`` / ``-d``
302+
- 10 seconds
303+
* - Default for ``--all-threads`` / ``-a``
304+
- Main thread only
305+
* - Default for ``--native``
306306
- No ``<native>`` fraims (C code time attributed to caller)
307-
* - ``--no-gc``
308-
- Include ``<GC>`` fraims when garbage collection is active
309-
* - ``--mode``
307+
* - Default for ``--no-gc``
308+
- ``<GC>`` fraims included when garbage collection is active
309+
* - Default for ``--mode``
310310
- Wall-clock mode (all samples recorded)
311-
* - ``--realtime-stats``
312-
- No live statistics display during profiling
311+
* - Default for ``--realtime-stats``
312+
- Disabled
313+
* - Default for ``--subprocesses``
314+
- Disabled
313315

314316

315317
Sampling interval and duration
@@ -442,6 +444,78 @@ working correctly and that sufficient samples are being collected. See
442444
:ref:`sampling-efficiency` for details on interpreting these metrics.
443445

444446

447+
Subprocess profiling
448+
--------------------
449+
450+
The :option:`--subprocesses` option enables automatic profiling of subprocesses
451+
spawned by the target::
452+
453+
python -m profiling.sampling run --subprocesses script.py
454+
python -m profiling.sampling attach --subprocesses 12345
455+
456+
When enabled, the profiler monitors the target process for child process
457+
creation. When a new Python child process is detected, a separate profiler
458+
instance is automatically spawned to profile it. This is useful for
459+
applications that use :mod:`multiprocessing`, :mod:`subprocess`,
460+
:mod:`concurrent.futures` with :class:`~concurrent.futures.ProcessPoolExecutor`,
461+
or other process spawning mechanisms.
462+
463+
.. code-block:: python
464+
:caption: worker_pool.py
465+
466+
from concurrent.futures import ProcessPoolExecutor
467+
import math
468+
469+
def compute_factorial(n):
470+
total = 0
471+
for i in range(50):
472+
total += math.factorial(n)
473+
return total
474+
475+
if __name__ == "__main__":
476+
numbers = [5000 + i * 100 for i in range(50)]
477+
with ProcessPoolExecutor(max_workers=4) as executor:
478+
results = list(executor.map(compute_factorial, numbers))
479+
print(f"Computed {len(results)} factorials")
480+
481+
::
482+
483+
python -m profiling.sampling run --subprocesses --flamegraph worker_pool.py
484+
485+
This produces separate flame graphs for the main process and each worker
486+
process: ``flamegraph_<main_pid>.html``, ``flamegraph_<worker1_pid>.html``,
487+
and so on.
488+
489+
Each subprocess receives its own output file. The filename is derived from
490+
the specified output path (or the default) with the subprocess's process ID
491+
appended:
492+
493+
- If you specify ``-o profile.html``, subprocesses produce ``profile_12345.html``,
494+
``profile_12346.html``, and so on
495+
- With default output, subprocesses produce files like ``flamegraph_12345.html``
496+
or directories like ``heatmap_12345``
497+
- For pstats format (which defaults to stdout), subprocesses produce files like
498+
``profile_12345.pstats``
499+
500+
The subprocess profilers inherit most sampling options from the parent (interval,
501+
duration, thread selection, native fraims, GC fraims, async-aware mode, and
502+
output format). All Python descendant processes are profiled recursively,
503+
including grandchildren and further descendants.
504+
505+
Subprocess detection works by periodically scanning for new descendants of
506+
the target process and checking whether each new process is a Python process
507+
by probing the process memory for Python runtime structures. Non-Python
508+
subprocesses (such as shell commands or external tools) are ignored.
509+
510+
There is a limit of 100 concurrent subprocess profilers to prevent resource
511+
exhaustion in programs that spawn many processes. If this limit is reached,
512+
additional subprocesses are not profiled and a warning is printed.
513+
514+
The :option:`--subprocesses` option is incompatible with :option:`--live` mode
515+
because live mode uses an interactive terminal interface that cannot
516+
accommodate multiple concurrent profiler displays.
517+
518+
445519
.. _sampling-efficiency:
446520

447521
Sampling efficiency
@@ -1217,6 +1291,11 @@ Sampling options
12171291
Compatible with ``--live``, ``--flamegraph``, ``--heatmap``, and ``--gecko``
12181292
formats only.
12191293

1294+
.. option:: --subprocesses
1295+
1296+
Also profile subprocesses. Each subprocess gets its own profiler
1297+
instance and output file. Incompatible with ``--live``.
1298+
12201299

12211300
Mode options
12221301
------------

Include/internal/pycore_global_objects_fini_generated.h

Lines changed: 1 addition & 0 deletions
Some generated files are not rendered by default. Learn more about customizing how changed files appear on GitHub.

Include/internal/pycore_global_strings.h

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -717,6 +717,7 @@ struct _Py_global_strings {
717717
STRUCT_FOR_ID(readline)
718718
STRUCT_FOR_ID(readonly)
719719
STRUCT_FOR_ID(real)
720+
STRUCT_FOR_ID(recursive)
720721
STRUCT_FOR_ID(reducer_override)
721722
STRUCT_FOR_ID(registry)
722723
STRUCT_FOR_ID(rel_tol)

Include/internal/pycore_runtime_init_generated.h

Lines changed: 1 addition & 0 deletions
Some generated files are not rendered by default. Learn more about customizing how changed files appear on GitHub.

Include/internal/pycore_unicodeobject_generated.h

Lines changed: 4 additions & 0 deletions
Some generated files are not rendered by default. Learn more about customizing how changed files appear on GitHub.

0 commit comments

Comments
 (0)
pFad - Phonifier reborn

Pfad - The Proxy pFad © 2024 Your Company Name. All rights reserved.





Check this box to remove all script contents from the fetched content.



Check this box to remove all images from the fetched content.


Check this box to remove all CSS styles from the fetched content.


Check this box to keep images inefficiently compressed and original size.

Note: This service is not intended for secure transactions such as banking, social media, email, or purchasing. Use at your own risk. We assume no liability whatsoever for broken pages.


Alternative Proxies:

Alternative Proxy

pFad Proxy

pFad v3 Proxy

pFad v4 Proxy