From 4afc4f37c70eb3b275cdcbd99818fb47493aa7ad Mon Sep 17 00:00:00 2001 From: Alexandru Gagniuc Date: Tue, 15 Dec 2020 13:15:43 -0600 Subject: doc: FIT image: Clarify format and simplify syntax ** Introduction There are currently four ways to load an OS image with u-boot 1. SPL -> u-boot -> bootm 2. SPL blue falcon mode 3. "Basic" FIT image (CONFIG_LOAD_FIT) 4. "Full-featured" FIT image (CONFIG_LOAD_FIT_FULL) These four code paths were developed independently, and share very little code. (3) and (4), behave very differently, are littered with special cases. They even have different DTS syntax and properties. The cause of this divergence is that the FIT format specification leaves a number of things open to interpretation. The purpose of this change is to enable the reduction of code size, duplication, and complexity by updating and streamlining the FIT format. We are only marginally concerned with backwards compatibility, because we don't have inter-compatibility. For example, CONFIG_LOAD_FIT is able to load images that CONFIG_LOAD_FIT_FULL won't. This is a direct result of the incompatible syntax between the two implementations. Ideally, these changes would enable "simple" FIT to be a subset of the "full" fit implementation, and share most code. These changes should also eliminate the need for falcon mode (although we are not advocating for the removal of falcon mode at this time). ** Description of changes * The "configurations" node is now mandatory Guessing how to load components based on their "os" and "type" invites confusion and superfluous heuristics. Instead, require each FIT image to be explicit on how components should be loaded. * Eliminate "ramdisk", "setup", "standalone", and "fpga" properties Having too many special purpose properties requires special-casing FIT loading code. When a special property can be handled by another property, it is redundant. - A "ramdisk" is identical to a loadable. Thus ramdisk images should be placed under "loadables". - A "setup" node can be achieved by using a "kernel" or "firmware" property instead. - "standalone" is used for u-boot nodes. The correct property to use in this case is "firmware". - "fpga" is a loadable * Prioritize control between "firmware" and "kernel" "firmware" and "kernel" are special nodes in that control is passed to the "entry-point" of the image. Both can be present, for example, an OP-TEE firmware with a linux kernel. When both are present, control is passed to the "firmware" image. ** Further generalizations (not included herein) The "firmware" and "kernel" properties could be generalized as a "next-boot-stage", or similar name. This "next" stage would be special in that it is both executable, and is the stage that is passed control. For example, "next-stage" could be an op-tee image, with linux as a loadable, or a u-boot image. Signed-off-by: Alexandru Gagniuc --- doc/uImage.FIT/source_file_format.txt | 33 ++++++++++++++------------------- 1 file changed, 14 insertions(+), 19 deletions(-) diff --git a/doc/uImage.FIT/source_file_format.txt b/doc/uImage.FIT/source_file_format.txt index 633f227c59c..00ed3ebe935 100644 --- a/doc/uImage.FIT/source_file_format.txt +++ b/doc/uImage.FIT/source_file_format.txt @@ -126,12 +126,10 @@ Root node of the uImage Tree should have the following layout: load addresses supplied within sub-image nodes. May be omitted when no entry or load addresses are used. - Mandatory node: + Mandatory nodes: - images : This node contains a set of sub-nodes, each of them representing single component sub-image (like kernel, ramdisk, etc.). At least one sub-image is required. - - Optional node: - configurations : Contains a set of available configuration nodes and defines a default configuration. @@ -169,8 +167,8 @@ the '/images' node should have the following layout: to "none". Conditionally mandatory property: - - os : OS name, mandatory for types "kernel" and "ramdisk". Valid OS names - are: "openbsd", "netbsd", "freebsd", "4_4bsd", "linux", "svr4", "esix", + - os : OS name, mandatory for types "kernel". Valid OS names are: + "openbsd", "netbsd", "freebsd", "4_4bsd", "linux", "svr4", "esix", "solaris", "irix", "sco", "dell", "ncr", "lynxos", "vxworks", "psos", "qnx", "u-boot", "rtems", "unity", "integrity". - arch : Architecture name, mandatory for types: "standalone", "kernel", @@ -179,10 +177,13 @@ the '/images' node should have the following layout: "sparc64", "m68k", "microblaze", "nios2", "blackfin", "avr32", "st200", "sandbox". - entry : entry point address, address size is determined by - '#address-cells' property of the root node. Mandatory for for types: - "standalone" and "kernel". + '#address-cells' property of the root node. + Mandatory for types: "firmware", and "kernel". - load : load address, address size is determined by '#address-cells' - property of the root node. Mandatory for types: "standalone" and "kernel". + property of the root node. + Mandatory for types: "firmware", and "kernel". + - compatible : compatible method for loading image. + Mandatory for types: "fpga", and images that do not specify a load address. Optional nodes: - hash-1 : Each hash sub-node represents separate hash or checksum @@ -205,9 +206,8 @@ o hash-1 6) '/configurations' node ------------------------- -The 'configurations' node is optional. If present, it allows to create a -convenient, labeled boot configurations, which combine together kernel images -with their ramdisks and fdt blobs. +The 'configurations' node creates convenient, labeled boot configurations, +which combine together kernel images with their ramdisks and fdt blobs. The 'configurations' node has has the following structure: @@ -236,27 +236,22 @@ Each configuration has the following structure: o config-1 |- description = "configuration description" |- kernel = "kernel sub-node unit name" - |- ramdisk = "ramdisk sub-node unit name" |- fdt = "fdt sub-node unit-name" [, "fdt overlay sub-node unit-name", ...] - |- fpga = "fpga sub-node unit-name" |- loadables = "loadables sub-node unit-name" |- compatible = "vendor,board-style device tree compatible string" Mandatory properties: - description : Textual configuration description. - - kernel : Unit name of the corresponding kernel image (image sub-node of a - "kernel" type). + - kernel or firmware: Unit name of the corresponding kernel or firmware + (u-boot, op-tee, etc) image. If both "kernel" and "firmware" are specified, + control is passed to the firmware image. Optional properties: - - ramdisk : Unit name of the corresponding ramdisk image (component image - node of a "ramdisk" type). - fdt : Unit name of the corresponding fdt blob (component image node of a "fdt type"). Additional fdt overlay nodes can be supplied which signify that the resulting device tree blob is generated by the first base fdt blob with all subsequent overlays applied. - - setup : Unit name of the corresponding setup binary (used for booting - an x86 kernel). This contains the setup.bin file built by the kernel. - fpga : Unit name of the corresponding fpga bitstream blob (component image node of a "fpga type"). - loadables : Unit name containing a list of additional binaries to be -- cgit v1.3.1 From 10a1df3cd43ba2fe893d5dd1aeb2e7109ef0007f Mon Sep 17 00:00:00 2001 From: Heinrich Schuchardt Date: Fri, 1 Jan 2021 01:21:11 +0100 Subject: doc: update Kernel documentation build system Update the docomentation build system according to Linux v5.11-rc1. With this patch we can build the HTML documentation using either of Sphinx 2 and Sphinx 3. Signed-off-by: Heinrich Schuchardt Reviewed-by: Simon Glass [Fix warning in b4860qds.rst about "Switch Settings"] Signed-off-by: Tom Rini --- doc/board/freescale/b4860qds.rst | 8 +- doc/conf.py | 141 +++++++++--- doc/sphinx/automarkup.py | 290 ++++++++++++++++++++++++ doc/sphinx/cdomain.py | 93 +++++++- doc/sphinx/kernel_abi.py | 194 ++++++++++++++++ doc/sphinx/kernel_feat.py | 169 ++++++++++++++ doc/sphinx/kerneldoc.py | 15 +- doc/sphinx/kernellog.py | 6 +- doc/sphinx/kfigure.py | 6 +- doc/sphinx/load_config.py | 27 ++- doc/sphinx/maintainers_include.py | 197 +++++++++++++++++ doc/sphinx/parallel-wrapper.sh | 33 +++ doc/sphinx/parse-headers.pl | 6 +- doc/sphinx/requirements.txt | 5 +- scripts/kernel-doc | 450 ++++++++++++++++++++++++++++---------- 15 files changed, 1480 insertions(+), 160 deletions(-) create mode 100644 doc/sphinx/automarkup.py create mode 100644 doc/sphinx/kernel_abi.py create mode 100644 doc/sphinx/kernel_feat.py create mode 100755 doc/sphinx/maintainers_include.py create mode 100644 doc/sphinx/parallel-wrapper.sh diff --git a/doc/board/freescale/b4860qds.rst b/doc/board/freescale/b4860qds.rst index de14d857b91..6b5b1218d48 100644 --- a/doc/board/freescale/b4860qds.rst +++ b/doc/board/freescale/b4860qds.rst @@ -133,8 +133,8 @@ B4420 has: B4860QDS Default Settings ------------------------- -Switch Settings -^^^^^^^^^^^^^^^ +B4860QDS Switch Settings +^^^^^^^^^^^^^^^^^^^^^^^^ .. code-block:: none @@ -166,8 +166,8 @@ NOR boot:: B4420QDS Default Settings ------------------------- -Switch Settings -^^^^^^^^^^^^^^^ +B4420QDS Switch Settings +^^^^^^^^^^^^^^^^^^^^^^^^ .. code-block:: none diff --git a/doc/conf.py b/doc/conf.py index ee7f2017242..1b9f3591d5a 100644 --- a/doc/conf.py +++ b/doc/conf.py @@ -16,6 +16,8 @@ import sys import os import sphinx +from subprocess import check_output + # Get Sphinx version major, minor, patch = sphinx.version_info[:3] @@ -31,39 +33,98 @@ from load_config import loadConfig # If your documentation needs a minimal Sphinx version, state it here. needs_sphinx = '1.3' -latex_engine = 'xelatex' - # Add any Sphinx extension module names here, as strings. They can be # extensions coming with Sphinx (named 'sphinx.ext.*') or your custom # ones. -extensions = ['kerneldoc', 'rstFlatTable', 'kernel_include', 'kfigure'] +extensions = ['kerneldoc', 'rstFlatTable', 'kernel_include', + 'kfigure', 'sphinx.ext.ifconfig', 'automarkup', + 'maintainers_include', 'sphinx.ext.autosectionlabel', + 'kernel_abi', 'kernel_feat'] # -# cdomain is badly broken in Sphinx 3+. Leaving it out generates *most* -# of the docs correctly, but not all. +# cdomain is badly broken in Sphinx 3+. Leaving it out generates *most* +# of the docs correctly, but not all. Scream bloody murder but allow +# the process to proceed; hopefully somebody will fix this properly soon. # if major >= 3: + sys.stderr.write('''WARNING: The kernel documentation build process + support for Sphinx v3.0 and above is brand new. Be prepared for + possible issues in the generated output. + ''') if (major > 3) or (minor > 0 or patch >= 2): - sys.stderr.write('''The build process with Sphinx 3+ is broken. -You will have to remove -W in doc/Makefile. -''') # Sphinx c function parser is more pedantic with regards to type # checking. Due to that, having macros at c:function cause problems. - # Those needed to be escaped by using c_id_attributes[] array + # Those needed to be scaped by using c_id_attributes[] array c_id_attributes = [ - - # include/linux/compiler.h + # GCC Compiler types not parsed by Sphinx: + "__restrict__", + + # include/linux/compiler_types.h: + "__iomem", + "__kernel", + "noinstr", + "notrace", + "__percpu", + "__rcu", + "__user", + + # include/linux/compiler_attributes.h: + "__alias", + "__aligned", + "__aligned_largest", + "__always_inline", + "__assume_aligned", + "__cold", + "__attribute_const__", + "__copy", + "__pure", + "__designated_init", + "__visible", + "__printf", + "__scanf", + "__gnu_inline", + "__malloc", + "__mode", + "__no_caller_saved_registers", + "__noclone", + "__nonstring", + "__noreturn", + "__packed", + "__pure", + "__section", + "__always_unused", "__maybe_unused", + "__used", + "__weak", + "noinline", # include/efi.h "EFIAPI", # include/efi_loader.h "__efi_runtime", + + # include/linux/memblock.h: + "__init_memblock", + "__meminit", + + # include/linux/init.h: + "__init", + "__ref", + + # include/linux/linkage.h: + "asmlinkage", ] else: extensions.append('cdomain') + if major == 1 and minor < 7: + sys.stderr.write('WARNING: Sphinx 1.7 or greater will be required as of ' + 'the v2021.04 release\n') + +# Ensure that autosectionlabel will produce unique names +autosectionlabel_prefix_document = True +autosectionlabel_maxdepth = 2 # The name of the math extension changed on Sphinx 1.4 if (major == 1 and minor > 3) or (major > 1): @@ -86,9 +147,9 @@ source_suffix = '.rst' master_doc = 'index' # General information about the project. -project = 'Das U-Boot' -copyright = 'The U-Boot development community' -author = 'The U-Boot development community' +project = 'The Linux Kernel' +copyright = 'The kernel development community' +author = 'The kernel development community' # The version info for the project you're documenting, acts as replacement for # |version| and |release|, also used in various other places throughout the @@ -199,7 +260,7 @@ except ImportError: # The name of an image file (relative to this directory) to place at the top # of the sidebar. -html_logo = '../tools/logos/u-boot_logo.svg' +#html_logo = None # The name of an image file (within the static path) to use as favicon of the # docs. This file should be a Windows icon file (.ico) being 16x16 or 32x32 @@ -229,7 +290,7 @@ html_context = { # If true, SmartyPants will be used to convert quotes and dashes to # typographically correct entities. -#html_use_smartypants = True +html_use_smartypants = False # Custom sidebar templates, maps document names to template names. #html_sidebars = {} @@ -279,7 +340,7 @@ html_context = { #html_search_scorer = 'scorer.js' # Output file base name for HTML help builder. -htmlhelp_basename = 'TheUBootdoc' +htmlhelp_basename = 'TheLinuxKerneldoc' # -- Options for LaTeX output --------------------------------------------- @@ -288,7 +349,7 @@ latex_elements = { 'papersize': 'a4paper', # The font size ('10pt', '11pt' or '12pt'). -'pointsize': '8pt', +'pointsize': '11pt', # Latex figure (float) alignment #'figure_align': 'htbp', @@ -301,13 +362,24 @@ latex_elements = { 'preamble': ''' % Use some font with UTF-8 support with XeLaTeX \\usepackage{fontspec} - \\setsansfont{DejaVu Serif} - \\setromanfont{DejaVu Sans} + \\setsansfont{DejaVu Sans} + \\setromanfont{DejaVu Serif} \\setmonofont{DejaVu Sans Mono} - ''' } +# At least one book (translations) may have Asian characters +# with are only displayed if xeCJK is used + +cjk_cmd = check_output(['fc-list', '--format="%{family[0]}\n"']).decode('utf-8', 'ignore') +if cjk_cmd.find("Noto Sans CJK SC") >= 0: + print ("enabling CJK for LaTeX builder") + latex_elements['preamble'] += ''' + % This is needed for translations + \\usepackage{xeCJK} + \\setCJKmainfont{Noto Sans CJK SC} + ''' + # Fix reference escape troubles with Sphinx 1.4.x if major == 1 and minor > 3: latex_elements['preamble'] += '\\renewcommand*{\\DUrole}[2]{ #2 }\n' @@ -398,10 +470,23 @@ if major == 1 and minor < 6: # author, documentclass [howto, manual, or own class]). # Sorted in alphabetical order latex_documents = [ - ('index', 'u-boot-hacker-manual.tex', 'U-Boot Hacker Manual', - 'The U-Boot development community', 'manual'), ] +# Add all other index files from Documentation/ subdirectories +for fn in os.listdir('.'): + doc = os.path.join(fn, "index") + if os.path.exists(doc + ".rst"): + has = False + for l in latex_documents: + if l[0] == doc: + has = True + break + if not has: + latex_documents.append((doc, fn + '.tex', + 'Linux %s Documentation' % fn.capitalize(), + 'The kernel development community', + 'manual')) + # The name of an image file (relative to this directory) to place at the top of # the title page. #latex_logo = None @@ -428,7 +513,7 @@ latex_documents = [ # One entry per manual page. List of tuples # (source start file, name, description, authors, manual section). man_pages = [ - (master_doc, 'dasuboot', 'The U-Boot Documentation', + (master_doc, 'thelinuxkernel', 'The Linux Kernel Documentation', [author], 1) ] @@ -442,8 +527,8 @@ man_pages = [ # (source start file, target name, title, author, # dir menu entry, description, category) texinfo_documents = [ - (master_doc, 'DasUBoot', 'The U-Boot Documentation', - author, 'DasUBoot', 'One line description of project.', + (master_doc, 'TheLinuxKernel', 'The Linux Kernel Documentation', + author, 'TheLinuxKernel', 'One line description of project.', 'Miscellaneous'), ] @@ -535,13 +620,13 @@ epub_exclude_files = ['search.html'] # Grouping the document tree into PDF files. List of tuples # (source start file, target name, title, author, options). # -# See the Sphinx chapter of http://ralsina.me/static/manual.pdf +# See the Sphinx chapter of https://ralsina.me/static/manual.pdf # # FIXME: Do not add the index file here; the result will be too big. Adding # multiple PDF files here actually tries to get the cross-referencing right # *between* PDF files. pdf_documents = [ - ('uboot-documentation', u'U-Boot', u'U-Boot', u'J. Random Bozo'), + ('kernel-documentation', u'Kernel', u'Kernel', u'J. Random Bozo'), ] # kernel-doc extension configuration for running Sphinx directly (e.g. by Read diff --git a/doc/sphinx/automarkup.py b/doc/sphinx/automarkup.py new file mode 100644 index 00000000000..953b24b6e2b --- /dev/null +++ b/doc/sphinx/automarkup.py @@ -0,0 +1,290 @@ +# SPDX-License-Identifier: GPL-2.0 +# Copyright 2019 Jonathan Corbet +# +# Apply kernel-specific tweaks after the initial document processing +# has been done. +# +from docutils import nodes +import sphinx +from sphinx import addnodes +if sphinx.version_info[0] < 2 or \ + sphinx.version_info[0] == 2 and sphinx.version_info[1] < 1: + from sphinx.environment import NoUri +else: + from sphinx.errors import NoUri +import re +from itertools import chain + +# +# Python 2 lacks re.ASCII... +# +try: + ascii_p3 = re.ASCII +except AttributeError: + ascii_p3 = 0 + +# +# Regex nastiness. Of course. +# Try to identify "function()" that's not already marked up some +# other way. Sphinx doesn't like a lot of stuff right after a +# :c:func: block (i.e. ":c:func:`mmap()`s" flakes out), so the last +# bit tries to restrict matches to things that won't create trouble. +# +RE_function = re.compile(r'\b(([a-zA-Z_]\w+)\(\))', flags=ascii_p3) + +# +# Sphinx 2 uses the same :c:type role for struct, union, enum and typedef +# +RE_generic_type = re.compile(r'\b(struct|union|enum|typedef)\s+([a-zA-Z_]\w+)', + flags=ascii_p3) + +# +# Sphinx 3 uses a different C role for each one of struct, union, enum and +# typedef +# +RE_struct = re.compile(r'\b(struct)\s+([a-zA-Z_]\w+)', flags=ascii_p3) +RE_union = re.compile(r'\b(union)\s+([a-zA-Z_]\w+)', flags=ascii_p3) +RE_enum = re.compile(r'\b(enum)\s+([a-zA-Z_]\w+)', flags=ascii_p3) +RE_typedef = re.compile(r'\b(typedef)\s+([a-zA-Z_]\w+)', flags=ascii_p3) + +# +# Detects a reference to a documentation page of the form Documentation/... with +# an optional extension +# +RE_doc = re.compile(r'\bDocumentation(/[\w\-_/]+)(\.\w+)*') + +RE_namespace = re.compile(r'^\s*..\s*c:namespace::\s*(\S+)\s*$') + +# +# Reserved C words that we should skip when cross-referencing +# +Skipnames = [ 'for', 'if', 'register', 'sizeof', 'struct', 'unsigned' ] + + +# +# Many places in the docs refer to common system calls. It is +# pointless to try to cross-reference them and, as has been known +# to happen, somebody defining a function by these names can lead +# to the creation of incorrect and confusing cross references. So +# just don't even try with these names. +# +Skipfuncs = [ 'open', 'close', 'read', 'write', 'fcntl', 'mmap', + 'select', 'poll', 'fork', 'execve', 'clone', 'ioctl', + 'socket' ] + +c_namespace = '' + +def markup_refs(docname, app, node): + t = node.astext() + done = 0 + repl = [ ] + # + # Associate each regex with the function that will markup its matches + # + markup_func_sphinx2 = {RE_doc: markup_doc_ref, + RE_function: markup_c_ref, + RE_generic_type: markup_c_ref} + + markup_func_sphinx3 = {RE_doc: markup_doc_ref, + RE_function: markup_func_ref_sphinx3, + RE_struct: markup_c_ref, + RE_union: markup_c_ref, + RE_enum: markup_c_ref, + RE_typedef: markup_c_ref} + + if sphinx.version_info[0] >= 3: + markup_func = markup_func_sphinx3 + else: + markup_func = markup_func_sphinx2 + + match_iterators = [regex.finditer(t) for regex in markup_func] + # + # Sort all references by the starting position in text + # + sorted_matches = sorted(chain(*match_iterators), key=lambda m: m.start()) + for m in sorted_matches: + # + # Include any text prior to match as a normal text node. + # + if m.start() > done: + repl.append(nodes.Text(t[done:m.start()])) + + # + # Call the function associated with the regex that matched this text and + # append its return to the text + # + repl.append(markup_func[m.re](docname, app, m)) + + done = m.end() + if done < len(t): + repl.append(nodes.Text(t[done:])) + return repl + +# +# In sphinx3 we can cross-reference to C macro and function, each one with its +# own C role, but both match the same regex, so we try both. +# +def markup_func_ref_sphinx3(docname, app, match): + class_str = ['c-func', 'c-macro'] + reftype_str = ['function', 'macro'] + + cdom = app.env.domains['c'] + # + # Go through the dance of getting an xref out of the C domain + # + base_target = match.group(2) + target_text = nodes.Text(match.group(0)) + xref = None + possible_targets = [base_target] + # Check if this document has a namespace, and if so, try + # cross-referencing inside it first. + if c_namespace: + possible_targets.insert(0, c_namespace + "." + base_target) + + if base_target not in Skipnames: + for target in possible_targets: + if target not in Skipfuncs: + for class_s, reftype_s in zip(class_str, reftype_str): + lit_text = nodes.literal(classes=['xref', 'c', class_s]) + lit_text += target_text + pxref = addnodes.pending_xref('', refdomain = 'c', + reftype = reftype_s, + reftarget = target, modname = None, + classname = None) + # + # XXX The Latex builder will throw NoUri exceptions here, + # work around that by ignoring them. + # + try: + xref = cdom.resolve_xref(app.env, docname, app.builder, + reftype_s, target, pxref, + lit_text) + except NoUri: + xref = None + + if xref: + return xref + + return target_text + +def markup_c_ref(docname, app, match): + class_str = {# Sphinx 2 only + RE_function: 'c-func', + RE_generic_type: 'c-type', + # Sphinx 3+ only + RE_struct: 'c-struct', + RE_union: 'c-union', + RE_enum: 'c-enum', + RE_typedef: 'c-type', + } + reftype_str = {# Sphinx 2 only + RE_function: 'function', + RE_generic_type: 'type', + # Sphinx 3+ only + RE_struct: 'struct', + RE_union: 'union', + RE_enum: 'enum', + RE_typedef: 'type', + } + + cdom = app.env.domains['c'] + # + # Go through the dance of getting an xref out of the C domain + # + base_target = match.group(2) + target_text = nodes.Text(match.group(0)) + xref = None + possible_targets = [base_target] + # Check if this document has a namespace, and if so, try + # cross-referencing inside it first. + if c_namespace: + possible_targets.insert(0, c_namespace + "." + base_target) + + if base_target not in Skipnames: + for target in possible_targets: + if not (match.re == RE_function and target in Skipfuncs): + lit_text = nodes.literal(classes=['xref', 'c', class_str[match.re]]) + lit_text += target_text + pxref = addnodes.pending_xref('', refdomain = 'c', + reftype = reftype_str[match.re], + reftarget = target, modname = None, + classname = None) + # + # XXX The Latex builder will throw NoUri exceptions here, + # work around that by ignoring them. + # + try: + xref = cdom.resolve_xref(app.env, docname, app.builder, + reftype_str[match.re], target, pxref, + lit_text) + except NoUri: + xref = None + + if xref: + return xref + + return target_text + +# +# Try to replace a documentation reference of the form Documentation/... with a +# cross reference to that page +# +def markup_doc_ref(docname, app, match): + stddom = app.env.domains['std'] + # + # Go through the dance of getting an xref out of the std domain + # + target = match.group(1) + xref = None + pxref = addnodes.pending_xref('', refdomain = 'std', reftype = 'doc', + reftarget = target, modname = None, + classname = None, refexplicit = False) + # + # XXX The Latex builder will throw NoUri exceptions here, + # work around that by ignoring them. + # + try: + xref = stddom.resolve_xref(app.env, docname, app.builder, 'doc', + target, pxref, None) + except NoUri: + xref = None + # + # Return the xref if we got it; otherwise just return the plain text. + # + if xref: + return xref + else: + return nodes.Text(match.group(0)) + +def get_c_namespace(app, docname): + source = app.env.doc2path(docname) + with open(source) as f: + for l in f: + match = RE_namespace.search(l) + if match: + return match.group(1) + return '' + +def auto_markup(app, doctree, name): + global c_namespace + c_namespace = get_c_namespace(app, name) + # + # This loop could eventually be improved on. Someday maybe we + # want a proper tree traversal with a lot of awareness of which + # kinds of nodes to prune. But this works well for now. + # + # The nodes.literal test catches ``literal text``, its purpose is to + # avoid adding cross-references to functions that have been explicitly + # marked with cc:func:. + # + for para in doctree.traverse(nodes.paragraph): + for node in para.traverse(nodes.Text): + if not isinstance(node.parent, nodes.literal): + node.parent.replace(node, markup_refs(name, app, node)) + +def setup(app): + app.connect('doctree-resolved', auto_markup) + return { + 'parallel_read_safe': True, + 'parallel_write_safe': True, + } diff --git a/doc/sphinx/cdomain.py b/doc/sphinx/cdomain.py index cbac8e608dc..014a5229e57 100644 --- a/doc/sphinx/cdomain.py +++ b/doc/sphinx/cdomain.py @@ -40,14 +40,94 @@ from sphinx import addnodes from sphinx.domains.c import c_funcptr_sig_re, c_sig_re from sphinx.domains.c import CObject as Base_CObject from sphinx.domains.c import CDomain as Base_CDomain +from itertools import chain +import re -__version__ = '1.0' +__version__ = '1.1' # Get Sphinx version major, minor, patch = sphinx.version_info[:3] +# Namespace to be prepended to the full name +namespace = None + +# +# Handle trivial newer c domain tags that are part of Sphinx 3.1 c domain tags +# - Store the namespace if ".. c:namespace::" tag is found +# +RE_namespace = re.compile(r'^\s*..\s*c:namespace::\s*(\S+)\s*$') + +def markup_namespace(match): + global namespace + + namespace = match.group(1) + + return "" + +# +# Handle c:macro for function-style declaration +# +RE_macro = re.compile(r'^\s*..\s*c:macro::\s*(\S+)\s+(\S.*)\s*$') +def markup_macro(match): + return ".. c:function:: " + match.group(1) + ' ' + match.group(2) + +# +# Handle newer c domain tags that are evaluated as .. c:type: for +# backward-compatibility with Sphinx < 3.0 +# +RE_ctype = re.compile(r'^\s*..\s*c:(struct|union|enum|enumerator|alias)::\s*(.*)$') + +def markup_ctype(match): + return ".. c:type:: " + match.group(2) + +# +# Handle newer c domain tags that are evaluated as :c:type: for +# backward-compatibility with Sphinx < 3.0 +# +RE_ctype_refs = re.compile(r':c:(var|struct|union|enum|enumerator)::`([^\`]+)`') +def markup_ctype_refs(match): + return ":c:type:`" + match.group(2) + '`' + +# +# Simply convert :c:expr: and :c:texpr: into a literal block. +# +RE_expr = re.compile(r':c:(expr|texpr):`([^\`]+)`') +def markup_c_expr(match): + return '\ ``' + match.group(2) + '``\ ' + +# +# Parse Sphinx 3.x C markups, replacing them by backward-compatible ones +# +def c_markups(app, docname, source): + result = "" + markup_func = { + RE_namespace: markup_namespace, + RE_expr: markup_c_expr, + RE_macro: markup_macro, + RE_ctype: markup_ctype, + RE_ctype_refs: markup_ctype_refs, + } + + lines = iter(source[0].splitlines(True)) + for n in lines: + match_iterators = [regex.finditer(n) for regex in markup_func] + matches = sorted(chain(*match_iterators), key=lambda m: m.start()) + for m in matches: + n = n[:m.start()] + markup_func[m.re](m) + n[m.end():] + + result = result + n + + source[0] = result + +# +# Now implements support for the cdomain namespacing logic +# + def setup(app): + # Handle easy Sphinx 3.1+ simple new tags: :c:expr and .. c:namespace:: + app.connect('source-read', c_markups) + if (major == 1 and minor < 8): app.override_domain(CDomain) else: @@ -75,6 +155,8 @@ class CObject(Base_CObject): function-like macro, the name of the macro is returned. Otherwise ``False`` is returned. """ + global namespace + if not self.objtype == 'function': return False @@ -107,11 +189,16 @@ class CObject(Base_CObject): param += nodes.emphasis(argname, argname) paramlist += param + if namespace: + fullname = namespace + "." + fullname + return fullname def handle_signature(self, sig, signode): """Transform a C signature into RST nodes.""" + global namespace + fullname = self.handle_func_like_macro(sig, signode) if not fullname: fullname = super(CObject, self).handle_signature(sig, signode) @@ -122,6 +209,10 @@ class CObject(Base_CObject): else: # FIXME: handle :name: value of other declaration types? pass + else: + if namespace: + fullname = namespace + "." + fullname + return fullname def add_target_and_index(self, name, sig, signode): diff --git a/doc/sphinx/kernel_abi.py b/doc/sphinx/kernel_abi.py new file mode 100644 index 00000000000..f3da859c987 --- /dev/null +++ b/doc/sphinx/kernel_abi.py @@ -0,0 +1,194 @@ +# -*- coding: utf-8; mode: python -*- +# coding=utf-8 +# SPDX-License-Identifier: GPL-2.0 +# +u""" + kernel-abi + ~~~~~~~~~~ + + Implementation of the ``kernel-abi`` reST-directive. + + :copyright: Copyright (C) 2016 Markus Heiser + :copyright: Copyright (C) 2016-2020 Mauro Carvalho Chehab + :maintained-by: Mauro Carvalho Chehab + :license: GPL Version 2, June 1991 see Linux/COPYING for details. + + The ``kernel-abi`` (:py:class:`KernelCmd`) directive calls the + scripts/get_abi.pl script to parse the Kernel ABI files. + + Overview of directive's argument and options. + + .. code-block:: rst + + .. kernel-abi:: + :debug: + + The argument ```` is required. It contains the + location of the ABI files to be parsed. + + ``debug`` + Inserts a code-block with the *raw* reST. Sometimes it is helpful to see + what reST is generated. + +""" + +import codecs +import os +import subprocess +import sys +import re +import kernellog + +from os import path + +from docutils import nodes, statemachine +from docutils.statemachine import ViewList +from docutils.parsers.rst import directives, Directive +from docutils.utils.error_reporting import ErrorString + +# +# AutodocReporter is only good up to Sphinx 1.7 +# +import sphinx + +Use_SSI = sphinx.__version__[:3] >= '1.7' +if Use_SSI: + from sphinx.util.docutils import switch_source_input +else: + from sphinx.ext.autodoc import AutodocReporter + +__version__ = '1.0' + +def setup(app): + + app.add_directive("kernel-abi", KernelCmd) + return dict( + version = __version__ + , parallel_read_safe = True + , parallel_write_safe = True + ) + +class KernelCmd(Directive): + + u"""KernelABI (``kernel-abi``) directive""" + + required_arguments = 1 + optional_arguments = 2 + has_content = False + final_argument_whitespace = True + + option_spec = { + "debug" : directives.flag, + "rst" : directives.unchanged + } + + def run(self): + + doc = self.state.document + if not doc.settings.file_insertion_enabled: + raise self.warning("docutils: file insertion disabled") + + env = doc.settings.env + cwd = path.dirname(doc.current_source) + cmd = "get_abi.pl rest --enable-lineno --dir " + cmd += self.arguments[0] + + if 'rst' in self.options: + cmd += " --rst-source" + + srctree = path.abspath(os.environ["srctree"]) + + fname = cmd + + # extend PATH with $(srctree)/scripts + path_env = os.pathsep.join([ + srctree + os.sep + "scripts", + os.environ["PATH"] + ]) + shell_env = os.environ.copy() + shell_env["PATH"] = path_env + shell_env["srctree"] = srctree + + lines = self.runCmd(cmd, shell=True, cwd=cwd, env=shell_env) + nodeList = self.nestedParse(lines, self.arguments[0]) + return nodeList + + def runCmd(self, cmd, **kwargs): + u"""Run command ``cmd`` and return it's stdout as unicode.""" + + try: + proc = subprocess.Popen( + cmd + , stdout = subprocess.PIPE + , stderr = subprocess.PIPE + , **kwargs + ) + out, err = proc.communicate() + + out, err = codecs.decode(out, 'utf-8'), codecs.decode(err, 'utf-8') + + if proc.returncode != 0: + raise self.severe( + u"command '%s' failed with return code %d" + % (cmd, proc.returncode) + ) + except OSError as exc: + raise self.severe(u"problems with '%s' directive: %s." + % (self.name, ErrorString(exc))) + return out + + def nestedParse(self, lines, fname): + content = ViewList() + node = nodes.section() + + if "debug" in self.options: + code_block = "\n\n.. code-block:: rst\n :linenos:\n" + for l in lines.split("\n"): + code_block += "\n " + l + lines = code_block + "\n\n" + + line_regex = re.compile("^#define LINENO (\S+)\#([0-9]+)$") + ln = 0 + n = 0 + f = fname + + for line in lines.split("\n"): + n = n + 1 + match = line_regex.search(line) + if match: + new_f = match.group(1) + + # Sphinx parser is lazy: it stops parsing contents in the + # middle, if it is too big. So, handle it per input file + if new_f != f and content: + self.do_parse(content, node) + content = ViewList() + + f = new_f + + # sphinx counts lines from 0 + ln = int(match.group(2)) - 1 + else: + content.append(line, f, ln) + + kernellog.info(self.state.document.settings.env.app, "%s: parsed %i lines" % (fname, n)) + + if content: + self.do_parse(content, node) + + return node.children + + def do_parse(self, content, node): + if Use_SSI: + with switch_source_input(self.state, content): + self.state.nested_parse(content, 0, node, match_titles=1) + else: + buf = self.state.memo.title_styles, self.state.memo.section_level, self.state.memo.reporter + + self.state.memo.title_styles = [] + self.state.memo.section_level = 0 + self.state.memo.reporter = AutodocReporter(content, self.state.memo.reporter) + try: + self.state.nested_parse(content, 0, node, match_titles=1) + finally: + self.state.memo.title_styles, self.state.memo.section_level, self.state.memo.reporter = buf diff --git a/doc/sphinx/kernel_feat.py b/doc/sphinx/kernel_feat.py new file mode 100644 index 00000000000..2fee04f1ded --- /dev/null +++ b/doc/sphinx/kernel_feat.py @@ -0,0 +1,169 @@ +# coding=utf-8 +# SPDX-License-Identifier: GPL-2.0 +# +u""" + kernel-feat + ~~~~~~~~~~~ + + Implementation of the ``kernel-feat`` reST-directive. + + :copyright: Copyright (C) 2016 Markus Heiser + :copyright: Copyright (C) 2016-2019 Mauro Carvalho Chehab + :maintained-by: Mauro Carvalho Chehab + :license: GPL Version 2, June 1991 see Linux/COPYING for details. + + The ``kernel-feat`` (:py:class:`KernelFeat`) directive calls the + scripts/get_feat.pl script to parse the Kernel ABI files. + + Overview of directive's argument and options. + + .. code-block:: rst + + .. kernel-feat:: + :debug: + + The argument ```` is required. It contains the + location of the ABI files to be parsed. + + ``debug`` + Inserts a code-block with the *raw* reST. Sometimes it is helpful to see + what reST is generated. + +""" + +import codecs +import os +import subprocess +import sys + +from os import path + +from docutils import nodes, statemachine +from docutils.statemachine import ViewList +from docutils.parsers.rst import directives, Directive +from docutils.utils.error_reporting import ErrorString + +# +# AutodocReporter is only good up to Sphinx 1.7 +# +import sphinx + +Use_SSI = sphinx.__version__[:3] >= '1.7' +if Use_SSI: + from sphinx.util.docutils import switch_source_input +else: + from sphinx.ext.autodoc import AutodocReporter + +__version__ = '1.0' + +def setup(app): + + app.add_directive("kernel-feat", KernelFeat) + return dict( + version = __version__ + , parallel_read_safe = True + , parallel_write_safe = True + ) + +class KernelFeat(Directive): + + u"""KernelFeat (``kernel-feat``) directive""" + + required_arguments = 1 + optional_arguments = 2 + has_content = False + final_argument_whitespace = True + + option_spec = { + "debug" : directives.flag + } + + def warn(self, message, **replace): + replace["fname"] = self.state.document.current_source + replace["line_no"] = replace.get("line_no", self.lineno) + message = ("%(fname)s:%(line_no)s: [kernel-feat WARN] : " + message) % replace + self.state.document.settings.env.app.warn(message, prefix="") + + def run(self): + + doc = self.state.document + if not doc.settings.file_insertion_enabled: + raise self.warning("docutils: file insertion disabled") + + env = doc.settings.env + cwd = path.dirname(doc.current_source) + cmd = "get_feat.pl rest --dir " + cmd += self.arguments[0] + + if len(self.arguments) > 1: + cmd += " --arch " + self.arguments[1] + + srctree = path.abspath(os.environ["srctree"]) + + fname = cmd + + # extend PATH with $(srctree)/scripts + path_env = os.pathsep.join([ + srctree + os.sep + "scripts", + os.environ["PATH"] + ]) + shell_env = os.environ.copy() + shell_env["PATH"] = path_env + shell_env["srctree"] = srctree + + lines = self.runCmd(cmd, shell=True, cwd=cwd, env=shell_env) + nodeList = self.nestedParse(lines, fname) + return nodeList + + def runCmd(self, cmd, **kwargs): + u"""Run command ``cmd`` and return it's stdout as unicode.""" + + try: + proc = subprocess.Popen( + cmd + , stdout = subprocess.PIPE + , stderr = subprocess.PIPE + , **kwargs + ) + out, err = proc.communicate() + + out, err = codecs.decode(out, 'utf-8'), codecs.decode(err, 'utf-8') + + if proc.returncode != 0: + raise self.severe( + u"command '%s' failed with return code %d" + % (cmd, proc.returncode) + ) + except OSError as exc: + raise self.severe(u"problems with '%s' directive: %s." + % (self.name, ErrorString(exc))) + return out + + def nestedParse(self, lines, fname): + content = ViewList() + node = nodes.section() + + if "debug" in self.options: + code_block = "\n\n.. code-block:: rst\n :linenos:\n" + for l in lines.split("\n"): + code_block += "\n " + l + lines = code_block + "\n\n" + + for c, l in enumerate(lines.split("\n")): + content.append(l, fname, c) + + buf = self.state.memo.title_styles, self.state.memo.section_level, self.state.memo.reporter + + if Use_SSI: + with switch_source_input(self.state, content): + self.state.nested_parse(content, 0, node, match_titles=1) + else: + self.state.memo.title_styles = [] + self.state.memo.section_level = 0 + self.state.memo.reporter = AutodocReporter(content, self.state.memo.reporter) + try: + self.state.nested_parse(content, 0, node, match_titles=1) + finally: + self.state.memo.title_styles, self.state.memo.section_level, self.state.memo.reporter = buf + + return node.children diff --git a/doc/sphinx/kerneldoc.py b/doc/sphinx/kerneldoc.py index 4bcbd6ae01c..e9857ab904f 100644 --- a/doc/sphinx/kerneldoc.py +++ b/doc/sphinx/kerneldoc.py @@ -62,6 +62,7 @@ class KernelDocDirective(Directive): 'export': directives.unchanged, 'internal': directives.unchanged, 'identifiers': directives.unchanged, + 'no-identifiers': directives.unchanged, 'functions': directives.unchanged, } has_content = False @@ -70,6 +71,11 @@ class KernelDocDirective(Directive): env = self.state.document.settings.env cmd = [env.config.kerneldoc_bin, '-rst', '-enable-lineno'] + # Pass the version string to kernel-doc, as it needs to use a different + # dialect, depending what the C domain supports for each specific + # Sphinx versions + cmd += ['-sphinx-version', sphinx.__version__] + filename = env.config.kerneldoc_srctree + '/' + self.arguments[0] export_file_patterns = [] @@ -99,6 +105,12 @@ class KernelDocDirective(Directive): else: cmd += ['-no-doc-sections'] + if 'no-identifiers' in self.options: + no_identifiers = self.options.get('no-identifiers').split() + if no_identifiers: + for i in no_identifiers: + cmd += ['-nosymbol', i] + for pattern in export_file_patterns: for f in glob.glob(env.config.kerneldoc_srctree + '/' + pattern): env.note_dependency(os.path.abspath(f)) @@ -136,7 +148,8 @@ class KernelDocDirective(Directive): lineoffset = int(match.group(1)) - 1 # we must eat our comments since the upset the markup else: - result.append(line, filename, lineoffset) + doc = env.srcdir + "/" + env.docname + ":" + str(self.lineno) + result.append(line, doc + ": " + filename, lineoffset) lineoffset += 1 node = nodes.section() diff --git a/doc/sphinx/kernellog.py b/doc/sphinx/kernellog.py index af924f51a7d..8ac7d274f54 100644 --- a/doc/sphinx/kernellog.py +++ b/doc/sphinx/kernellog.py @@ -25,4 +25,8 @@ def verbose(app, message): else: app.verbose(message) - +def info(app, message): + if UseLogging: + logger.info(message) + else: + app.info(message) diff --git a/doc/sphinx/kfigure.py b/doc/sphinx/kfigure.py index fbfe6693bb6..788704886ee 100644 --- a/doc/sphinx/kfigure.py +++ b/doc/sphinx/kfigure.py @@ -29,7 +29,7 @@ u""" Used tools: - * ``dot(1)``: Graphviz (http://www.graphviz.org). If Graphviz is not + * ``dot(1)``: Graphviz (https://www.graphviz.org). If Graphviz is not available, the DOT language is inserted as literal-block. * SVG to PDF: To generate PDF, you need at least one of this tools: @@ -41,7 +41,7 @@ u""" * generate PDF from SVG / used by PDF (LaTeX) builder * generate SVG (html-builder) and PDF (latex-builder) from DOT files. - DOT: see http://www.graphviz.org/content/dot-language + DOT: see https://www.graphviz.org/content/dot-language """ @@ -182,7 +182,7 @@ def setupTools(app): kernellog.verbose(app, "use dot(1) from: " + dot_cmd) else: kernellog.warn(app, "dot(1) not found, for better output quality install " - "graphviz from http://www.graphviz.org") + "graphviz from https://www.graphviz.org") if convert_cmd: kernellog.verbose(app, "use convert(1) from: " + convert_cmd) else: diff --git a/doc/sphinx/load_config.py b/doc/sphinx/load_config.py index 301a21aa4f6..eeb394b39e2 100644 --- a/doc/sphinx/load_config.py +++ b/doc/sphinx/load_config.py @@ -21,6 +21,29 @@ def loadConfig(namespace): and os.path.normpath(namespace["__file__"]) != os.path.normpath(config_file) ): config_file = os.path.abspath(config_file) + # Let's avoid one conf.py file just due to latex_documents + start = config_file.find('Documentation/') + if start >= 0: + start = config_file.find('/', start + 1) + + end = config_file.rfind('/') + if start >= 0 and end > 0: + dir = config_file[start + 1:end] + + print("source directory: %s" % dir) + new_latex_docs = [] + latex_documents = namespace['latex_documents'] + + for l in latex_documents: + if l[0].find(dir + '/') == 0: + has = True + fn = l[0][len(dir) + 1:] + new_latex_docs.append((fn, l[1], l[2], l[3], l[4])) + break + + namespace['latex_documents'] = new_latex_docs + + # If there is an extra conf.py file, load it if os.path.isfile(config_file): sys.stdout.write("load additional sphinx-config: %s\n" % config_file) config = namespace.copy() @@ -29,4 +52,6 @@ def loadConfig(namespace): del config['__file__'] namespace.update(config) else: - sys.stderr.write("WARNING: additional sphinx-config not found: %s\n" % config_file) + config = namespace.copy() + config['tags'].add("subproject") + namespace.update(config) diff --git a/doc/sphinx/maintainers_include.py b/doc/sphinx/maintainers_include.py new file mode 100755 index 00000000000..dc8fed48d3c --- /dev/null +++ b/doc/sphinx/maintainers_include.py @@ -0,0 +1,197 @@ +#!/usr/bin/env python +# SPDX-License-Identifier: GPL-2.0 +# -*- coding: utf-8; mode: python -*- +# pylint: disable=R0903, C0330, R0914, R0912, E0401 + +u""" + maintainers-include + ~~~~~~~~~~~~~~~~~~~ + + Implementation of the ``maintainers-include`` reST-directive. + + :copyright: Copyright (C) 2019 Kees Cook + :license: GPL Version 2, June 1991 see linux/COPYING for details. + + The ``maintainers-include`` reST-directive performs extensive parsing + specific to the Linux kernel's standard "MAINTAINERS" file, in an + effort to avoid needing to heavily mark up the original plain text. +""" + +import sys +import re +import os.path + +from docutils import statemachine +from docutils.utils.error_reporting import ErrorString +from docutils.parsers.rst import Directive +from docutils.parsers.rst.directives.misc import Include + +__version__ = '1.0' + +def setup(app): + app.add_directive("maintainers-include", MaintainersInclude) + return dict( + version = __version__, + parallel_read_safe = True, + parallel_write_safe = True + ) + +class MaintainersInclude(Include): + u"""MaintainersInclude (``maintainers-include``) directive""" + required_arguments = 0 + + def parse_maintainers(self, path): + """Parse all the MAINTAINERS lines into ReST for human-readability""" + + result = list() + result.append(".. _maintainers:") + result.append("") + + # Poor man's state machine. + descriptions = False + maintainers = False + subsystems = False + + # Field letter to field name mapping. + field_letter = None + fields = dict() + + prev = None + field_prev = "" + field_content = "" + + for line in open(path): + if sys.version_info.major == 2: + line = unicode(line, 'utf-8') + # Have we reached the end of the preformatted Descriptions text? + if descriptions and line.startswith('Maintainers'): + descriptions = False + # Ensure a blank line following the last "|"-prefixed line. + result.append("") + + # Start subsystem processing? This is to skip processing the text + # between the Maintainers heading and the first subsystem name. + if maintainers and not subsystems: + if re.search('^[A-Z0-9]', line): + subsystems = True + + # Drop needless input whitespace. + line = line.rstrip() + + # Linkify all non-wildcard refs to ReST files in Documentation/. + pat = '(Documentation/([^\s\?\*]*)\.rst)' + m = re.search(pat, line) + if m: + # maintainers.rst is in a subdirectory, so include "../". + line = re.sub(pat, ':doc:`%s <../%s>`' % (m.group(2), m.group(2)), line) + + # Check state machine for output rendering behavior. + output = None + if descriptions: + # Escape the escapes in preformatted text. + output = "| %s" % (line.replace("\\", "\\\\")) + # Look for and record field letter to field name mappings: + # R: Designated *reviewer*: FullName + m = re.search("\s(\S):\s", line) + if m: + field_letter = m.group(1) + if field_letter and not field_letter in fields: + m = re.search("\*([^\*]+)\*", line) + if m: + fields[field_letter] = m.group(1) + elif subsystems: + # Skip empty lines: subsystem parser adds them as needed. + if len(line) == 0: + continue + # Subsystem fields are batched into "field_content" + if line[1] != ':': + # Render a subsystem entry as: + # SUBSYSTEM NAME + # ~~~~~~~~~~~~~~ + + # Flush pending field content. + output = field_content + "\n\n" + field_content = "" + + # Collapse whitespace in subsystem name. + heading = re.sub("\s+", " ", line) + output = output + "%s\n%s" % (heading, "~" * len(heading)) + field_prev = "" + else: + # Render a subsystem field as: + # :Field: entry + # entry... + field, details = line.split(':', 1) + details = details.strip() + + # Mark paths (and regexes) as literal text for improved + # readability and to escape any escapes. + if field in ['F', 'N', 'X', 'K']: + # But only if not already marked :) + if not ':doc:' in details: + details = '``%s``' % (details) + + # Comma separate email field continuations. + if field == field_prev and field_prev in ['M', 'R', 'L']: + field_content = field_content + "," + + # Do not repeat field names, so that field entries + # will be collapsed together. + if field != field_prev: + output = field_content + "\n" + field_content = ":%s:" % (fields.get(field, field)) + field_content = field_content + "\n\t%s" % (details) + field_prev = field + else: + output = line + + # Re-split on any added newlines in any above parsing. + if output != None: + for separated in output.split('\n'): + result.append(separated) + + # Update the state machine when we find heading separators. + if line.startswith('----------'): + if prev.startswith('Descriptions'): + descriptions = True + if prev.startswith('Maintainers'): + maintainers = True + + # Retain previous line for state machine transitions. + prev = line + + # Flush pending field contents. + if field_content != "": + for separated in field_content.split('\n'): + result.append(separated) + + output = "\n".join(result) + # For debugging the pre-rendered results... + #print(output, file=open("/tmp/MAINTAINERS.rst", "w")) + + self.state_machine.insert_input( + statemachine.string2lines(output), path) + + def run(self): + """Include the MAINTAINERS file as part of this reST file.""" + if not self.state.document.settings.file_insertion_enabled: + raise self.warning('"%s" directive disabled.' % self.name) + + # Walk up source path directories to find Documentation/../ + path = self.state_machine.document.attributes['source'] + path = os.path.realpath(path) + tail = path + while tail != "Documentation" and tail != "": + (path, tail) = os.path.split(path) + + # Append "MAINTAINERS" + path = os.path.join(path, "MAINTAINERS") + + try: + self.state.document.settings.record_dependencies.add(path) + lines = self.parse_maintainers(path) + except IOError as error: + raise self.severe('Problems with "%s" directive path:\n%s.' % + (self.name, ErrorString(error))) + + return [] diff --git a/doc/sphinx/parallel-wrapper.sh b/doc/sphinx/parallel-wrapper.sh new file mode 100644 index 00000000000..e54c44ce117 --- /dev/null +++ b/doc/sphinx/parallel-wrapper.sh @@ -0,0 +1,33 @@ +#!/bin/sh +# SPDX-License-Identifier: GPL-2.0+ +# +# Figure out if we should follow a specific parallelism from the make +# environment (as exported by scripts/jobserver-exec), or fall back to +# the "auto" parallelism when "-jN" is not specified at the top-level +# "make" invocation. + +sphinx="$1" +shift || true + +parallel="$PARALLELISM" +if [ -z "$parallel" ] ; then + # If no parallelism is specified at the top-level make, then + # fall back to the expected "-jauto" mode that the "htmldocs" + # target has had. + auto=$(perl -e 'open IN,"'"$sphinx"' --version 2>&1 |"; + while () { + if (m/([\d\.]+)/) { + print "auto" if ($1 >= "1.7") + } + } + close IN') + if [ -n "$auto" ] ; then + parallel="$auto" + fi +fi +# Only if some parallelism has been determined do we add the -jN option. +if [ -n "$parallel" ] ; then + parallel="-j$parallel" +fi + +exec "$sphinx" $parallel "$@" diff --git a/doc/sphinx/parse-headers.pl b/doc/sphinx/parse-headers.pl index c518050ffc3..b063f2f1cfb 100755 --- a/doc/sphinx/parse-headers.pl +++ b/doc/sphinx/parse-headers.pl @@ -1,4 +1,4 @@ -#!/usr/bin/perl +#!/usr/bin/env perl use strict; use Text::Tabs; use Getopt::Long; @@ -110,7 +110,7 @@ while () { ) { my $s = $1; - $structs{$s} = "struct :c:type:`$s`\\ "; + $structs{$s} = "struct $s\\ "; next; } } @@ -393,7 +393,7 @@ Report bugs to Mauro Carvalho Chehab Copyright (c) 2016 by Mauro Carvalho Chehab . -License GPLv2: GNU GPL version 2 . +License GPLv2: GNU GPL version 2 . This is free software: you are free to change and redistribute it. There is NO WARRANTY, to the extent permitted by law. diff --git a/doc/sphinx/requirements.txt b/doc/sphinx/requirements.txt index 742be3e1261..5030d346d23 100644 --- a/doc/sphinx/requirements.txt +++ b/doc/sphinx/requirements.txt @@ -1,3 +1,4 @@ -docutils==0.12 -Sphinx==1.4.9 +docutils +Sphinx==2.4.4 sphinx_rtd_theme +six diff --git a/scripts/kernel-doc b/scripts/kernel-doc index 516cf1db32c..6dda8cb51f2 100755 --- a/scripts/kernel-doc +++ b/scripts/kernel-doc @@ -56,6 +56,13 @@ Output format selection (mutually exclusive): -rst Output reStructuredText format. -none Do not output documentation, only warnings. +Output format selection modifier (affects only ReST output): + + -sphinx-version Use the ReST C domain dialect compatible with an + specific Sphinx Version. + If not specified, kernel-doc will auto-detect using + the sphinx-build version found on PATH. + Output selection (mutually exclusive): -export Only output documentation for symbols that have been exported using EXPORT_SYMBOL() or EXPORT_SYMBOL_GPL() @@ -66,9 +73,8 @@ Output selection (mutually exclusive): -function NAME Only output documentation for the given function(s) or DOC: section title(s). All other functions and DOC: sections are ignored. May be specified multiple times. - -nofunction NAME Do NOT output documentation for the given function(s); - only output documentation for the other functions and - DOC: sections. May be specified multiple times. + -nosymbol NAME Exclude the specified symbols from the output + documentation. May be specified multiple times. Output selection modifiers: -no-doc-sections Do not output DOC: sections. @@ -81,6 +87,7 @@ Output selection modifiers: Other parameters: -v Verbose output, more warnings and other information. -h Print this help. + -Werror Treat warnings as errors. EOF print $message; @@ -213,7 +220,9 @@ my $type_constant = '\b``([^\`]+)``\b'; my $type_constant2 = '\%([-_\w]+)'; my $type_func = '(\w+)\(\)'; my $type_param = '\@(\w*((\.\w+)|(->\w+))*(\.\.\.)?)'; +my $type_param_ref = '([\!]?)\@(\w*((\.\w+)|(->\w+))*(\.\.\.)?)'; my $type_fp_param = '\@(\w+)\(\)'; # Special RST handling for func ptr params +my $type_fp_param2 = '\@(\w+->\S+)\(\)'; # Special RST handling for structs with func ptr params my $type_env = '(\$\w+)'; my $type_enum = '\&(enum\s*([_\w]+))'; my $type_struct = '\&(struct\s*([_\w]+))'; @@ -236,6 +245,7 @@ my @highlights_man = ( [$type_typedef, "\\\\fI\$1\\\\fP"], [$type_union, "\\\\fI\$1\\\\fP"], [$type_param, "\\\\fI\$1\\\\fP"], + [$type_param_ref, "\\\\fI\$1\$2\\\\fP"], [$type_member, "\\\\fI\$1\$2\$3\\\\fP"], [$type_fallback, "\\\\fI\$1\\\\fP"] ); @@ -249,14 +259,15 @@ my @highlights_rst = ( [$type_member_func, "\\:c\\:type\\:`\$1\$2\$3\\\\(\\\\) <\$1>`"], [$type_member, "\\:c\\:type\\:`\$1\$2\$3 <\$1>`"], [$type_fp_param, "**\$1\\\\(\\\\)**"], - [$type_func, "\\:c\\:func\\:`\$1()`"], + [$type_fp_param2, "**\$1\\\\(\\\\)**"], + [$type_func, "\$1()"], [$type_enum, "\\:c\\:type\\:`\$1 <\$2>`"], [$type_struct, "\\:c\\:type\\:`\$1 <\$2>`"], [$type_typedef, "\\:c\\:type\\:`\$1 <\$2>`"], [$type_union, "\\:c\\:type\\:`\$1 <\$2>`"], # in rst this can refer to any type [$type_fallback, "\\:c\\:type\\:`\$1`"], - [$type_param, "**\$1**"] + [$type_param_ref, "**\$1\$2**"] ); my $blankline_rst = "\n"; @@ -266,9 +277,12 @@ if ($#ARGV == -1) { } my $kernelversion; +my ($sphinx_major, $sphinx_minor, $sphinx_patch); + my $dohighlight = ""; my $verbose = 0; +my $Werror = 0; my $output_mode = "rst"; my $output_preformatted = 0; my $no_doc_sections = 0; @@ -280,12 +294,11 @@ my $modulename = "Kernel API"; use constant { OUTPUT_ALL => 0, # output all symbols and doc sections OUTPUT_INCLUDE => 1, # output only specified symbols - OUTPUT_EXCLUDE => 2, # output everything except specified symbols - OUTPUT_EXPORTED => 3, # output exported symbols - OUTPUT_INTERNAL => 4, # output non-exported symbols + OUTPUT_EXPORTED => 2, # output exported symbols + OUTPUT_INTERNAL => 3, # output non-exported symbols }; my $output_selection = OUTPUT_ALL; -my $show_not_found = 0; +my $show_not_found = 0; # No longer used my @export_file_list; @@ -307,6 +320,7 @@ my $man_date = ('January', 'February', 'March', 'April', 'May', 'June', # CAVEAT EMPTOR! Some of the others I localised may not want to be, which # could cause "use of undefined value" or other bugs. my ($function, %function_table, %parametertypes, $declaration_purpose); +my %nosymbol_table = (); my $declaration_start_line; my ($type, $declaration_name, $return_type); my ($newsection, $newcontents, $prototype, $brcount, %source_map); @@ -315,9 +329,21 @@ if (defined($ENV{'KBUILD_VERBOSE'})) { $verbose = "$ENV{'KBUILD_VERBOSE'}"; } +if (defined($ENV{'KDOC_WERROR'})) { + $Werror = "$ENV{'KDOC_WERROR'}"; +} + +if (defined($ENV{'KCFLAGS'})) { + my $kcflags = "$ENV{'KCFLAGS'}"; + + if ($kcflags =~ /Werror/) { + $Werror = 1; + } +} + # Generated docbook code is inserted in a template at a point where # docbook v3.1 requires a non-zero sequence of RefEntry's; see: -# http://www.oasis-open.org/docbook/documentation/reference/html/refentry.html +# https://www.oasis-open.org/docbook/documentation/reference/html/refentry.html # We keep track of number of generated entries and generate a dummy # if needs be to ensure the expanded template can be postprocessed # into html. @@ -327,13 +353,14 @@ my $lineprefix=""; # Parser states use constant { - STATE_NORMAL => 0, # normal code - STATE_NAME => 1, # looking for function name - STATE_BODY_MAYBE => 2, # body - or maybe more description - STATE_BODY => 3, # the body of the comment - STATE_PROTO => 4, # scanning prototype - STATE_DOCBLOCK => 5, # documentation block - STATE_INLINE => 6, # gathering documentation outside main block + STATE_NORMAL => 0, # normal code + STATE_NAME => 1, # looking for function name + STATE_BODY_MAYBE => 2, # body - or maybe more description + STATE_BODY => 3, # the body of the comment + STATE_BODY_WITH_BLANK_LINE => 4, # the body, which has a blank line + STATE_PROTO => 5, # scanning prototype + STATE_DOCBLOCK => 6, # documentation block + STATE_INLINE => 7, # gathering doc outside main block }; my $state; my $in_doc_sect; @@ -413,10 +440,9 @@ while ($ARGV[0] =~ m/^--?(.*)/) { $output_selection = OUTPUT_INCLUDE; $function = shift @ARGV; $function_table{$function} = 1; - } elsif ($cmd eq "nofunction") { # output all except specific functions - $output_selection = OUTPUT_EXCLUDE; - $function = shift @ARGV; - $function_table{$function} = 1; + } elsif ($cmd eq "nosymbol") { # Exclude specific symbols + my $symbol = shift @ARGV; + $nosymbol_table{$symbol} = 1; } elsif ($cmd eq "export") { # only exported symbols $output_selection = OUTPUT_EXPORTED; %function_table = (); @@ -428,6 +454,8 @@ while ($ARGV[0] =~ m/^--?(.*)/) { push(@export_file_list, $file); } elsif ($cmd eq "v") { $verbose = 1; + } elsif ($cmd eq "Werror") { + $Werror = 1; } elsif (($cmd eq "h") || ($cmd eq "help")) { usage(); } elsif ($cmd eq 'no-doc-sections') { @@ -435,7 +463,24 @@ while ($ARGV[0] =~ m/^--?(.*)/) { } elsif ($cmd eq 'enable-lineno') { $enable_lineno = 1; } elsif ($cmd eq 'show-not-found') { - $show_not_found = 1; + $show_not_found = 1; # A no-op but don't fail + } elsif ($cmd eq "sphinx-version") { + my $ver_string = shift @ARGV; + if ($ver_string =~ m/^(\d+)(\.\d+)?(\.\d+)?/) { + $sphinx_major = $1; + if (defined($2)) { + $sphinx_minor = substr($2,1); + } else { + $sphinx_minor = 0; + } + if (defined($3)) { + $sphinx_patch = substr($3,1) + } else { + $sphinx_patch = 0; + } + } else { + die "Sphinx version should either major.minor or major.minor.patch format\n"; + } } else { # Unknown argument usage(); @@ -444,6 +489,51 @@ while ($ARGV[0] =~ m/^--?(.*)/) { # continue execution near EOF; +# The C domain dialect changed on Sphinx 3. So, we need to check the +# version in order to produce the right tags. +sub findprog($) +{ + foreach(split(/:/, $ENV{PATH})) { + return "$_/$_[0]" if(-x "$_/$_[0]"); + } +} + +sub get_sphinx_version() +{ + my $ver; + + my $cmd = "sphinx-build"; + if (!findprog($cmd)) { + my $cmd = "sphinx-build3"; + if (!findprog($cmd)) { + $sphinx_major = 1; + $sphinx_minor = 2; + $sphinx_patch = 0; + printf STDERR "Warning: Sphinx version not found. Using default (Sphinx version %d.%d.%d)\n", + $sphinx_major, $sphinx_minor, $sphinx_patch; + return; + } + } + + open IN, "$cmd --version 2>&1 |"; + while () { + if (m/^\s*sphinx-build\s+([\d]+)\.([\d\.]+)(\+\/[\da-f]+)?$/) { + $sphinx_major = $1; + $sphinx_minor = $2; + $sphinx_patch = $3; + last; + } + # Sphinx 1.2.x uses a different format + if (m/^\s*Sphinx.*\s+([\d]+)\.([\d\.]+)$/) { + $sphinx_major = $1; + $sphinx_minor = $2; + $sphinx_patch = $3; + last; + } + } + close IN; +} + # get kernel version from env sub get_kernel_version() { my $version = 'unknown kernel version'; @@ -510,11 +600,11 @@ sub dump_doc_section { return; } + return if (defined($nosymbol_table{$name})); + if (($output_selection == OUTPUT_ALL) || - ($output_selection == OUTPUT_INCLUDE && - defined($function_table{$name})) || - ($output_selection == OUTPUT_EXCLUDE && - !defined($function_table{$name}))) + (($output_selection == OUTPUT_INCLUDE) && + defined($function_table{$name}))) { dump_section($file, $name, $contents); output_blockhead({'sectionlist' => \@sectionlist, @@ -597,10 +687,10 @@ sub output_function_man(%) { $type = $args{'parametertypes'}{$parameter}; if ($type =~ m/([^\(]*\(\*)\s*\)\s*\(([^\)]*)\)/) { # pointer-to-function - print ".BI \"" . $parenth . $1 . "\" " . $parameter . " \") (" . $2 . ")" . $post . "\"\n"; + print ".BI \"" . $parenth . $1 . "\" " . " \") (" . $2 . ")" . $post . "\"\n"; } else { $type =~ s/([^\*])$/$1 /; - print ".BI \"" . $parenth . $type . "\" " . $parameter . " \"" . $post . "\"\n"; + print ".BI \"" . $parenth . $type . "\" " . " \"" . $post . "\"\n"; } $count++; $parenth = ""; @@ -740,6 +830,8 @@ sub output_blockhead_rst(%) { my ($parameter, $section); foreach $section (@{$args{'sectionlist'}}) { + next if (defined($nosymbol_table{$section})); + if ($output_selection != OUTPUT_INCLUDE) { print "**$section**\n\n"; } @@ -825,16 +917,37 @@ sub output_function_rst(%) { my ($parameter, $section); my $oldprefix = $lineprefix; my $start = ""; - - if ($args{'typedef'}) { - print ".. c:type:: ". $args{'function'} . "\n\n"; - print_lineno($declaration_start_line); - print " **Typedef**: "; - $lineprefix = ""; - output_highlight_rst($args{'purpose'}); - $start = "\n\n**Syntax**\n\n ``"; + my $is_macro = 0; + + if ($sphinx_major < 3) { + if ($args{'typedef'}) { + print ".. c:type:: ". $args{'function'} . "\n\n"; + print_lineno($declaration_start_line); + print " **Typedef**: "; + $lineprefix = ""; + output_highlight_rst($args{'purpose'}); + $start = "\n\n**Syntax**\n\n ``"; + $is_macro = 1; + } else { + print ".. c:function:: "; + } } else { - print ".. c:function:: "; + if ($args{'typedef'} || $args{'functiontype'} eq "") { + $is_macro = 1; + print ".. c:macro:: ". $args{'function'} . "\n\n"; + } else { + print ".. c:function:: "; + } + + if ($args{'typedef'}) { + print_lineno($declaration_start_line); + print " **Typedef**: "; + $lineprefix = ""; + output_highlight_rst($args{'purpose'}); + $start = "\n\n**Syntax**\n\n ``"; + } else { + print "``" if ($is_macro); + } } if ($args{'functiontype'} ne "") { $start .= $args{'functiontype'} . " " . $args{'function'} . " ("; @@ -853,15 +966,17 @@ sub output_function_rst(%) { if ($type =~ m/([^\(]*\(\*)\s*\)\s*\(([^\)]*)\)/) { # pointer-to-function - print $1 . $parameter . ") (" . $2; + print $1 . $parameter . ") (" . $2 . ")"; } else { - print $type . " " . $parameter; + print $type; } } - if ($args{'typedef'}) { - print ");``\n\n"; + if ($is_macro) { + print ")``\n\n"; } else { print ")\n\n"; + } + if (!$args{'typedef'}) { print_lineno($declaration_start_line); $lineprefix = " "; output_highlight_rst($args{'purpose'}); @@ -876,7 +991,7 @@ sub output_function_rst(%) { $type = $args{'parametertypes'}{$parameter}; if ($type ne "") { - print "``$type $parameter``\n"; + print "``$type``\n"; } else { print "``$parameter``\n"; } @@ -917,9 +1032,14 @@ sub output_enum_rst(%) { my ($parameter); my $oldprefix = $lineprefix; my $count; - my $name = "enum " . $args{'enum'}; - print "\n\n.. c:type:: " . $name . "\n\n"; + if ($sphinx_major < 3) { + my $name = "enum " . $args{'enum'}; + print "\n\n.. c:type:: " . $name . "\n\n"; + } else { + my $name = $args{'enum'}; + print "\n\n.. c:enum:: " . $name . "\n\n"; + } print_lineno($declaration_start_line); $lineprefix = " "; output_highlight_rst($args{'purpose'}); @@ -945,8 +1065,13 @@ sub output_typedef_rst(%) { my %args = %{$_[0]}; my ($parameter); my $oldprefix = $lineprefix; - my $name = "typedef " . $args{'typedef'}; + my $name; + if ($sphinx_major < 3) { + $name = "typedef " . $args{'typedef'}; + } else { + $name = $args{'typedef'}; + } print "\n\n.. c:type:: " . $name . "\n\n"; print_lineno($declaration_start_line); $lineprefix = " "; @@ -961,9 +1086,18 @@ sub output_struct_rst(%) { my %args = %{$_[0]}; my ($parameter); my $oldprefix = $lineprefix; - my $name = $args{'type'} . " " . $args{'struct'}; - print "\n\n.. c:type:: " . $name . "\n\n"; + if ($sphinx_major < 3) { + my $name = $args{'type'} . " " . $args{'struct'}; + print "\n\n.. c:type:: " . $name . "\n\n"; + } else { + my $name = $args{'struct'}; + if ($args{'type'} eq 'union') { + print "\n\n.. c:union:: " . $name . "\n\n"; + } else { + print "\n\n.. c:struct:: " . $name . "\n\n"; + } + } print_lineno($declaration_start_line); $lineprefix = " "; output_highlight_rst($args{'purpose'}); @@ -1022,12 +1156,14 @@ sub output_declaration { my $name = shift; my $functype = shift; my $func = "output_${functype}_$output_mode"; + + return if (defined($nosymbol_table{$name})); + if (($output_selection == OUTPUT_ALL) || (($output_selection == OUTPUT_INCLUDE || $output_selection == OUTPUT_EXPORTED) && defined($function_table{$name})) || - (($output_selection == OUTPUT_EXCLUDE || - $output_selection == OUTPUT_INTERNAL) && + ($output_selection == OUTPUT_INTERNAL && !($functype eq "function" && defined($function_table{$name})))) { &$func(@_); @@ -1062,7 +1198,7 @@ sub dump_struct($$) { my $x = shift; my $file = shift; - if ($x =~ /(struct|union)\s+(\w+)\s*\{(.*)\}(\s*(__packed|__aligned|__attribute__\s*\(\([a-z0-9,_\s\(\)]*\)\)))*/) { + if ($x =~ /(struct|union)\s+(\w+)\s*\{(.*)\}(\s*(__packed|__aligned|____cacheline_aligned_in_smp|____cacheline_aligned|__attribute__\s*\(\([a-z0-9,_\s\(\)]*\)\)))*/) { my $decl_type = $1; $declaration_name = $2; my $members = $3; @@ -1073,11 +1209,15 @@ sub dump_struct($$) { # strip comments: $members =~ s/\/\*.*?\*\///gos; # strip attributes - $members =~ s/\s*__attribute__\s*\(\([a-z0-9,_\*\s\(\)]*\)\)//gi; - $members =~ s/\s*__aligned\s*\([^;]*\)//gos; - $members =~ s/\s*__packed\s*//gos; - $members =~ s/\s*CRYPTO_MINALIGN_ATTR//gos; + $members =~ s/\s*__attribute__\s*\(\([a-z0-9,_\*\s\(\)]*\)\)/ /gi; + $members =~ s/\s*__aligned\s*\([^;]*\)/ /gos; + $members =~ s/\s*__packed\s*/ /gos; + $members =~ s/\s*CRYPTO_MINALIGN_ATTR/ /gos; + $members =~ s/\s*____cacheline_aligned_in_smp/ /gos; + $members =~ s/\s*____cacheline_aligned/ /gos; + # replace DECLARE_BITMAP + $members =~ s/__ETHTOOL_DECLARE_LINK_MODE_MASK\s*\(([^\)]+)\)/DECLARE_BITMAP($1, __ETHTOOL_LINK_MODE_MASK_NBITS)/gos; $members =~ s/DECLARE_BITMAP\s*\(([^,)]+),\s*([^,)]+)\)/unsigned long $1\[BITS_TO_LONGS($2)\]/gos; # replace DECLARE_HASHTABLE $members =~ s/DECLARE_HASHTABLE\s*\(([^,)]+),\s*([^,)]+)\)/unsigned long $1\[1 << (($2) - 1)\]/gos; @@ -1204,6 +1344,8 @@ sub show_warnings($$) { my $functype = shift; my $name = shift; + return 0 if (defined($nosymbol_table{$name})); + return 1 if ($output_selection == OUTPUT_ALL); if ($output_selection == OUTPUT_EXPORTED) { @@ -1227,27 +1369,28 @@ sub show_warnings($$) { return 0; } } - if ($output_selection == OUTPUT_EXCLUDE) { - if (!defined($function_table{$name})) { - return 1; - } else { - return 0; - } - } die("Please add the new output type at show_warnings()"); } sub dump_enum($$) { my $x = shift; my $file = shift; + my $members; + $x =~ s@/\*.*?\*/@@gos; # strip comments. # strip #define macros inside enums $x =~ s@#\s*((define|ifdef)\s+|endif)[^;]*;@@gos; - if ($x =~ /enum\s+(\w+)\s*\{(.*)\}/) { + if ($x =~ /typedef\s+enum\s*\{(.*)\}\s*(\w*)\s*;/) { + $declaration_name = $2; + $members = $1; + } elsif ($x =~ /enum\s+(\w*)\s*\{(.*)\}/) { $declaration_name = $1; - my $members = $2; + $members = $2; + } + + if ($members) { my %_members; $members =~ s/\s+$//; @@ -1282,27 +1425,31 @@ sub dump_enum($$) { 'sections' => \%sections, 'purpose' => $declaration_purpose }); - } - else { + } else { print STDERR "${file}:$.: error: Cannot parse enum!\n"; ++$errors; } } +my $typedef_type = qr { ((?:\s+[\w\*]+\b){1,8})\s* }x; +my $typedef_ident = qr { \*?\s*(\w\S+)\s* }x; +my $typedef_args = qr { \s*\((.*)\); }x; + +my $typedef1 = qr { typedef$typedef_type\($typedef_ident\)$typedef_args }x; +my $typedef2 = qr { typedef$typedef_type$typedef_ident$typedef_args }x; + sub dump_typedef($$) { my $x = shift; my $file = shift; $x =~ s@/\*.*?\*/@@gos; # strip comments. - # Parse function prototypes - if ($x =~ /typedef\s+(\w+)\s*\(\*\s*(\w\S+)\s*\)\s*\((.*)\);/ || - $x =~ /typedef\s+(\w+)\s*(\w\S+)\s*\s*\((.*)\);/) { - - # Function typedefs + # Parse function typedef prototypes + if ($x =~ $typedef1 || $x =~ $typedef2) { $return_type = $1; $declaration_name = $2; my $args = $3; + $return_type =~ s/^\s+//; create_parameterlist($args, ',', $file, $declaration_name); @@ -1369,25 +1516,27 @@ sub create_parameterlist($$$$) { foreach my $arg (split($splitter, $args)) { # strip comments $arg =~ s/\/\*.*\*\///; + $arg =~ s/\s*EFIAPI\s*/ /g; # strip leading/trailing spaces $arg =~ s/^\s*//; $arg =~ s/\s*$//; $arg =~ s/\s+/ /; + if ($arg =~ /^#/) { # Treat preprocessor directive as a typeless variable just to fill # corresponding data structures "correctly". Catch it later in # output_* subs. - push_parameter($arg, "", $file); + push_parameter($arg, "", "", $file); } elsif ($arg =~ m/\(.+\)\s*\(/) { # pointer-to-function $arg =~ tr/#/,/; - $arg =~ m/[^\(]+\([\w\s]*\*?\s*([\w\.]*)\s*\)/; + $arg =~ m/[^\(]+\(\s*\*?\s*([\w\.]*)\s*\)/; $param = $1; $type = $arg; $type =~ s/([^\(]+\(\*?)\s*$param/$1/; save_struct_actual($param); - push_parameter($param, $type, $file, $declaration_name); + push_parameter($param, $type, $arg, $file, $declaration_name); } elsif ($arg) { $arg =~ s/\s*:\s*/:/g; $arg =~ s/\s*\[/\[/g; @@ -1412,26 +1561,28 @@ sub create_parameterlist($$$$) { foreach $param (@args) { if ($param =~ m/^(\*+)\s*(.*)/) { save_struct_actual($2); - push_parameter($2, "$type $1", $file, $declaration_name); + + push_parameter($2, "$type $1", $arg, $file, $declaration_name); } elsif ($param =~ m/(.*?):(\d+)/) { if ($type ne "") { # skip unnamed bit-fields save_struct_actual($1); - push_parameter($1, "$type:$2", $file, $declaration_name) + push_parameter($1, "$type:$2", $arg, $file, $declaration_name) } } else { save_struct_actual($param); - push_parameter($param, $type, $file, $declaration_name); + push_parameter($param, $type, $arg, $file, $declaration_name); } } } } } -sub push_parameter($$$$) { +sub push_parameter($$$$$) { my $param = shift; my $type = shift; + my $org_arg = shift; my $file = shift; my $declaration_name = shift; @@ -1449,6 +1600,10 @@ sub push_parameter($$$$) { # handles unnamed variable parameters $param = "..."; } + elsif ($param =~ /\w\.\.\.$/) { + # for named variable parameters of the form `x...`, remove the dots + $param =~ s/\.\.\.$//; + } if (!defined $parameterdescs{$param} || $parameterdescs{$param} eq "") { $parameterdescs{$param} = "variable arguments"; } @@ -1491,8 +1646,8 @@ sub push_parameter($$$$) { # "[blah" in a parameter string; ###$param =~ s/\s*//g; push @parameterlist, $param; - $type =~ s/\s\s+/ /g; - $parametertypes{$param} = $type; + $org_arg =~ s/\s\s+/ /g; + $parametertypes{$param} = $org_arg; } sub check_sections($$$$$) { @@ -1566,6 +1721,8 @@ sub dump_function($$) { my $file = shift; my $noret = 0; + print_lineno($new_start_line); + $prototype =~ s/^static +//; $prototype =~ s/^extern +//; $prototype =~ s/^asmlinkage +//; @@ -1580,6 +1737,7 @@ sub dump_function($$) { $prototype =~ s/__must_check +//; $prototype =~ s/__weak +//; $prototype =~ s/__sched +//; + $prototype =~ s/__printf\s*\(\s*\d*\s*,\s*\d*\s*\) +//; my $define = $prototype =~ s/^#\s*define\s+//; #ak added $prototype =~ s/__attribute__\s*\(\( (?: @@ -1640,30 +1798,48 @@ sub dump_function($$) { return; } - my $prms = join " ", @parameterlist; - check_sections($file, $declaration_name, "function", $sectcheck, $prms); + my $prms = join " ", @parameterlist; + check_sections($file, $declaration_name, "function", $sectcheck, $prms); - # This check emits a lot of warnings at the moment, because many - # functions don't have a 'Return' doc section. So until the number - # of warnings goes sufficiently down, the check is only performed in - # verbose mode. - # TODO: always perform the check. - if ($verbose && !$noret) { - check_return_section($file, $declaration_name, $return_type); - } + # This check emits a lot of warnings at the moment, because many + # functions don't have a 'Return' doc section. So until the number + # of warnings goes sufficiently down, the check is only performed in + # verbose mode. + # TODO: always perform the check. + if ($verbose && !$noret) { + check_return_section($file, $declaration_name, $return_type); + } - output_declaration($declaration_name, - 'function', - {'function' => $declaration_name, - 'module' => $modulename, - 'functiontype' => $return_type, - 'parameterlist' => \@parameterlist, - 'parameterdescs' => \%parameterdescs, - 'parametertypes' => \%parametertypes, - 'sectionlist' => \@sectionlist, - 'sections' => \%sections, - 'purpose' => $declaration_purpose - }); + # The function parser can be called with a typedef parameter. + # Handle it. + if ($return_type =~ /typedef/) { + output_declaration($declaration_name, + 'function', + {'function' => $declaration_name, + 'typedef' => 1, + 'module' => $modulename, + 'functiontype' => $return_type, + 'parameterlist' => \@parameterlist, + 'parameterdescs' => \%parameterdescs, + 'parametertypes' => \%parametertypes, + 'sectionlist' => \@sectionlist, + 'sections' => \%sections, + 'purpose' => $declaration_purpose + }); + } else { + output_declaration($declaration_name, + 'function', + {'function' => $declaration_name, + 'module' => $modulename, + 'functiontype' => $return_type, + 'parameterlist' => \@parameterlist, + 'parameterdescs' => \%parameterdescs, + 'parametertypes' => \%parametertypes, + 'sectionlist' => \@sectionlist, + 'sections' => \%sections, + 'purpose' => $declaration_purpose + }); + } } sub reset_state { @@ -1758,6 +1934,11 @@ sub process_proto_function($$) { $prototype =~ s@/\*.*?\*/@@gos; # strip comments. $prototype =~ s@[\r\n]+@ @gos; # strip newlines/cr's. $prototype =~ s@^\s+@@gos; # strip leading spaces + + # Handle prototypes for function pointers like: + # int (*pcs_config)(struct foo) + $prototype =~ s@^(\S+\s+)\(\s*\*(\S+)\)@$1$2@gos; + if ($prototype =~ /SYSCALL_DEFINE/) { syscall_munge(); } @@ -1836,6 +2017,7 @@ sub process_export_file($) { while () { if (/$export_symbol/) { + next if (defined($nosymbol_table{$2})); $function_table{$2} = 1; } } @@ -1867,7 +2049,7 @@ sub process_name($$) { if (/$doc_block/o) { $state = STATE_DOCBLOCK; $contents = ""; - $new_start_line = $. + 1; + $new_start_line = $.; if ( $1 eq "" ) { $section = $section_intro; @@ -1935,6 +2117,25 @@ sub process_name($$) { sub process_body($$) { my $file = shift; + # Until all named variable macro parameters are + # documented using the bare name (`x`) rather than with + # dots (`x...`), strip the dots: + if ($section =~ /\w\.\.\.$/) { + $section =~ s/\.\.\.$//; + + if ($verbose) { + print STDERR "${file}:$.: warning: Variable macro arguments should be documented without dots\n"; + ++$warnings; + } + } + + if ($state == STATE_BODY_WITH_BLANK_LINE && /^\s*\*\s?\S/) { + dump_section($file, $section, $contents); + $section = $section_default; + $new_start_line = $.; + $contents = ""; + } + if (/$doc_sect/i) { # case insensitive for supported section names $newsection = $1; $newcontents = $2; @@ -1987,19 +2188,23 @@ sub process_body($$) { $prototype = ""; $state = STATE_PROTO; $brcount = 0; + $new_start_line = $. + 1; } elsif (/$doc_content/) { - # miguel-style comment kludge, look for blank lines after - # @parameter line to signify start of description if ($1 eq "") { - if ($section =~ m/^@/ || $section eq $section_context) { + if ($section eq $section_context) { dump_section($file, $section, $contents); $section = $section_default; $contents = ""; $new_start_line = $.; + $state = STATE_BODY; } else { + if ($section ne $section_default) { + $state = STATE_BODY_WITH_BLANK_LINE; + } else { + $state = STATE_BODY; + } $contents .= "\n"; } - $state = STATE_BODY; } elsif ($state == STATE_BODY_MAYBE) { # Continued declaration purpose chomp($declaration_purpose); @@ -2131,7 +2336,7 @@ sub process_file($) { $file = map_filename($orig_file); - if (!open(IN,"<$file")) { + if (!open(IN_FILE,"<$file")) { print STDERR "Error: Cannot open file $file\n"; ++$errors; return; @@ -2140,9 +2345,9 @@ sub process_file($) { $. = 1; $section_counter = 0; - while () { + while () { while (s/\\\s*$//) { - $_ .= ; + $_ .= ; } # Replace tabs by spaces while ($_ =~ s/\t+/' ' x (length($&) * 8 - length($`) % 8)/e) {}; @@ -2151,7 +2356,8 @@ sub process_file($) { process_normal(); } elsif ($state == STATE_NAME) { process_name($file, $_); - } elsif ($state == STATE_BODY || $state == STATE_BODY_MAYBE) { + } elsif ($state == STATE_BODY || $state == STATE_BODY_MAYBE || + $state == STATE_BODY_WITH_BLANK_LINE) { process_body($file, $_); } elsif ($state == STATE_INLINE) { # scanning for inline parameters process_inline($file, $_); @@ -2163,17 +2369,24 @@ sub process_file($) { } # Make sure we got something interesting. - if ($initial_section_counter == $section_counter) { - if ($output_mode ne "none") { - print STDERR "${file}:1: warning: no structured comments found\n"; + if ($initial_section_counter == $section_counter && $ + output_mode ne "none") { + if ($output_selection == OUTPUT_INCLUDE) { + print STDERR "${file}:1: warning: '$_' not found\n" + for keys %function_table; } - if (($output_selection == OUTPUT_INCLUDE) && ($show_not_found == 1)) { - print STDERR " Was looking for '$_'.\n" for keys %function_table; + else { + print STDERR "${file}:1: warning: no structured comments found\n"; } } + close IN_FILE; } +if ($output_mode eq "rst") { + get_sphinx_version() if (!$sphinx_major); +} + $kernelversion = get_kernel_version(); # generate a sequence of code that will splice in highlighting information @@ -2220,4 +2433,9 @@ if ($verbose && $warnings) { print STDERR "$warnings warnings\n"; } -exit($output_mode eq "none" ? 0 : $errors); +if ($Werror && $warnings) { + print STDERR "$warnings warnings as Errors\n"; + exit($warnings); +} else { + exit($output_mode eq "none" ? 0 : $errors) +} -- cgit v1.3.1 From 9153924bc14c16836278faa9455053e9594a3970 Mon Sep 17 00:00:00 2001 From: Patrick Delaunay Date: Thu, 14 Jan 2021 12:04:04 +0100 Subject: doc: update help message Update the help message used for 'make help': Documentation targets: Linux kernel internal documentation in different formats from ReST: => U-Boot documentation in different formats from ReST: Signed-off-by: Patrick Delaunay Reviewed-by: Heinrich Schuchardt Reviewed-by: Bin Meng --- doc/Makefile | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/doc/Makefile b/doc/Makefile index 0e0da5666f8..a686d4728ec 100644 --- a/doc/Makefile +++ b/doc/Makefile @@ -106,7 +106,7 @@ cleandocs: $(Q)$(MAKE) BUILDDIR=$(abspath $(BUILDDIR)) $(build)=doc/media clean dochelp: - @echo ' Linux kernel internal documentation in different formats from ReST:' + @echo ' U-Boot documentation in different formats from ReST:' @echo ' htmldocs - HTML' @echo ' latexdocs - LaTeX' @echo ' pdfdocs - PDF' -- cgit v1.3.1 From 948991f764593de53855d665eee29dc55e9f0030 Mon Sep 17 00:00:00 2001 From: Heinrich Schuchardt Date: Mon, 18 Jan 2021 20:24:01 +0100 Subject: doc: structure doc/develop/index.rst Provide sub-chapters for 'Develop U-Boot' Signed-off-by: Heinrich Schuchardt Reviewed-by: Simon Glass --- doc/develop/index.rst | 22 +++++++++++++++++++--- 1 file changed, 19 insertions(+), 3 deletions(-) diff --git a/doc/develop/index.rst b/doc/develop/index.rst index 0a7e204b343..b108df8e1b5 100644 --- a/doc/develop/index.rst +++ b/doc/develop/index.rst @@ -3,13 +3,29 @@ Develop U-Boot ============== +Implementation +-------------- .. toctree:: - :maxdepth: 2 + :maxdepth: 1 - coccinelle commands - crash_dumps global_data logging + +Debugging +--------- + +.. toctree:: + :maxdepth: 1 + + crash_dumps trace + +Testing +------- + +.. toctree:: + :maxdepth: 1 + + coccinelle -- cgit v1.3.1 From ca6583d4e08934e1de66c9291ded516e962ae8de Mon Sep 17 00:00:00 2001 From: Heinrich Schuchardt Date: Mon, 18 Jan 2021 20:24:02 +0100 Subject: doc: move test/README to HTML documentation Move test/README to the 'Develop U-Boot' chapter of the HTML documentation. Signed-off-by: Heinrich Schuchardt Reviewed-by: Simon Glass --- doc/develop/index.rst | 1 + doc/develop/testing.rst | 96 +++++++++++++++++++++++++++++++++++++++++++++++++ test/README | 96 ------------------------------------------------- 3 files changed, 97 insertions(+), 96 deletions(-) create mode 100644 doc/develop/testing.rst delete mode 100644 test/README diff --git a/doc/develop/index.rst b/doc/develop/index.rst index b108df8e1b5..ed2e73bf562 100644 --- a/doc/develop/index.rst +++ b/doc/develop/index.rst @@ -29,3 +29,4 @@ Testing :maxdepth: 1 coccinelle + testing diff --git a/doc/develop/testing.rst b/doc/develop/testing.rst new file mode 100644 index 00000000000..4bc9ca3a6ae --- /dev/null +++ b/doc/develop/testing.rst @@ -0,0 +1,96 @@ +Testing in U-Boot +================= + +U-Boot has a large amount of code. This file describes how this code is +tested and what tests you should write when adding a new feature. + + +Running tests +------------- + +To run most tests on sandbox, type this: + + make check + +in the U-Boot directory. Note that only the pytest suite is run using this +command. + +Some tests take ages to run. To run just the quick ones, type this: + + make qcheck + + +Sandbox +------- +U-Boot can be built as a user-space application (e.g. for Linux). This +allows test to be executed without needing target hardware. The 'sandbox' +target provides this feature and it is widely used in tests. + + +Pytest Suite +------------ + +Many tests are available using the pytest suite, in test/py. This can run +either on sandbox or on real hardware. It relies on the U-Boot console to +inject test commands and check the result. It is slower to run than C code, +but provides the ability to unify lots of tests and summarise their results. + +You can run the tests on sandbox with: + + ./test/py/test.py --bd sandbox --build + +This will produce HTML output in build-sandbox/test-log.html + +See test/py/README.md for more information about the pytest suite. + + +tbot +---- + +Tbot provides a way to execute tests on target hardware. It is intended for +trying out both U-Boot and Linux (and potentially other software) on a +number of boards automatically. It can be used to create a continuous test +environment. See http://www.tbot.tools for more information. + + +Ad-hoc tests +------------ + +There are several ad-hoc tests which run outside the pytest environment: + + test/fs - File system test (shell script) + test/image - FIT and legacy image tests (shell script and Python) + test/stdint - A test that stdint.h can be used in U-Boot (shell script) + trace - Test for the tracing feature (shell script) + +TODO: Move these into pytest. + + +When to write tests +------------------- + +If you add code to U-Boot without a test you are taking a risk. Even if you +perform thorough manual testing at the time of submission, it may break when +future changes are made to U-Boot. It may even break when applied to mainline, +if other changes interact with it. A good mindset is that untested code +probably doesn't work and should be deleted. + +You can assume that the Pytest suite will be run before patches are accepted +to mainline, so this provides protection against future breakage. + +On the other hand there is quite a bit of code that is not covered with tests, +or is covered sparingly. So here are some suggestions: + +- If you are adding a new uclass, add a sandbox driver and a test that uses it +- If you are modifying code covered by an existing test, add a new test case + to cover your changes +- If the code you are modifying has not tests, consider writing one. Even a + very basic test is useful, and may be picked up and enhanced by others. It + is much easier to add onto a test - writing a new large test can seem + daunting to most contributors. + + +Future work +----------- + +Converting existing shell scripts into pytest tests. diff --git a/test/README b/test/README deleted file mode 100644 index 4bc9ca3a6ae..00000000000 --- a/test/README +++ /dev/null @@ -1,96 +0,0 @@ -Testing in U-Boot -================= - -U-Boot has a large amount of code. This file describes how this code is -tested and what tests you should write when adding a new feature. - - -Running tests -------------- - -To run most tests on sandbox, type this: - - make check - -in the U-Boot directory. Note that only the pytest suite is run using this -command. - -Some tests take ages to run. To run just the quick ones, type this: - - make qcheck - - -Sandbox -------- -U-Boot can be built as a user-space application (e.g. for Linux). This -allows test to be executed without needing target hardware. The 'sandbox' -target provides this feature and it is widely used in tests. - - -Pytest Suite ------------- - -Many tests are available using the pytest suite, in test/py. This can run -either on sandbox or on real hardware. It relies on the U-Boot console to -inject test commands and check the result. It is slower to run than C code, -but provides the ability to unify lots of tests and summarise their results. - -You can run the tests on sandbox with: - - ./test/py/test.py --bd sandbox --build - -This will produce HTML output in build-sandbox/test-log.html - -See test/py/README.md for more information about the pytest suite. - - -tbot ----- - -Tbot provides a way to execute tests on target hardware. It is intended for -trying out both U-Boot and Linux (and potentially other software) on a -number of boards automatically. It can be used to create a continuous test -environment. See http://www.tbot.tools for more information. - - -Ad-hoc tests ------------- - -There are several ad-hoc tests which run outside the pytest environment: - - test/fs - File system test (shell script) - test/image - FIT and legacy image tests (shell script and Python) - test/stdint - A test that stdint.h can be used in U-Boot (shell script) - trace - Test for the tracing feature (shell script) - -TODO: Move these into pytest. - - -When to write tests -------------------- - -If you add code to U-Boot without a test you are taking a risk. Even if you -perform thorough manual testing at the time of submission, it may break when -future changes are made to U-Boot. It may even break when applied to mainline, -if other changes interact with it. A good mindset is that untested code -probably doesn't work and should be deleted. - -You can assume that the Pytest suite will be run before patches are accepted -to mainline, so this provides protection against future breakage. - -On the other hand there is quite a bit of code that is not covered with tests, -or is covered sparingly. So here are some suggestions: - -- If you are adding a new uclass, add a sandbox driver and a test that uses it -- If you are modifying code covered by an existing test, add a new test case - to cover your changes -- If the code you are modifying has not tests, consider writing one. Even a - very basic test is useful, and may be picked up and enhanced by others. It - is much easier to add onto a test - writing a new large test can seem - daunting to most contributors. - - -Future work ------------ - -Converting existing shell scripts into pytest tests. -- cgit v1.3.1 From 0157619d5c8beb9cf06b096c83330a312a9bf65d Mon Sep 17 00:00:00 2001 From: Heinrich Schuchardt Date: Mon, 18 Jan 2021 20:24:03 +0100 Subject: doc: move test/py/README.md to HTML documentation Convert test/py/README.md to restructured text and add it to the generated HTML documentation. Signed-off-by: Heinrich Schuchardt Reviewed-by: Simon Glass --- doc/develop/index.rst | 1 + doc/develop/py_testing.rst | 426 +++++++++++++++++++++++++++++++++++++++++++++ test/py/README.md | 389 ----------------------------------------- 3 files changed, 427 insertions(+), 389 deletions(-) create mode 100644 doc/develop/py_testing.rst delete mode 100644 test/py/README.md diff --git a/doc/develop/index.rst b/doc/develop/index.rst index ed2e73bf562..83eb1307edb 100644 --- a/doc/develop/index.rst +++ b/doc/develop/index.rst @@ -30,3 +30,4 @@ Testing coccinelle testing + py_testing diff --git a/doc/develop/py_testing.rst b/doc/develop/py_testing.rst new file mode 100644 index 00000000000..f71e837aa96 --- /dev/null +++ b/doc/develop/py_testing.rst @@ -0,0 +1,426 @@ +U-Boot pytest suite +=================== + +Introduction +------------ + +This tool aims to test U-Boot by executing U-Boot shell commands using the +console interface. A single top-level script exists to execute or attach to the +U-Boot console, run the entire script of tests against it, and summarize the +results. Advantages of this approach are: + +- Testing is performed in the same way a user or script would interact with + U-Boot; there can be no disconnect. +- There is no need to write or embed test-related code into U-Boot itself. + It is asserted that writing test-related code in Python is simpler and more + flexible than writing it all in C. +- It is reasonably simple to interact with U-Boot in this way. + +Requirements +------------ + +The test suite is implemented using pytest. Interaction with the U-Boot console +involves executing some binary and interacting with its stdin/stdout. You will +need to implement various "hook" scripts that are called by the test suite at +the appropriate time. + +In order to run the test suite at a minimum we require that both Python 3 and +pip for Python 3 are installed. All of the required python modules are +described in the requirements.txt file in the /test/py/ directory and can be +installed via the command + +.. code-block:: bash + + pip install -r requirements.txt + +In order to execute certain tests on their supported platforms other tools +will be required. The following is an incomplete list: + +* gdisk +* dfu-util +* dtc +* openssl +* sudo OR guestmount +* e2fsprogs +* util-linux +* coreutils +* dosfstools +* efitools +* mount +* mtools +* sbsigntool +* udisks2 + +Please use the appropriate commands for your distribution to match these tools +up with the package that provides them. + +The test script supports either: + +- Executing a sandbox port of U-Boot on the local machine as a sub-process, + and interacting with it over stdin/stdout. +- Executing an external "hook" scripts to flash a U-Boot binary onto a + physical board, attach to the board's console stream, and reset the board. + Further details are described later. + +Using `virtualenv` to provide requirements +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +The recommended way to run the test suite, in order to ensure reproducibility +is to use `virtualenv` to set up the necessary environment. This can be done +via the following commands: + + +.. code-block:: console + + $ cd /path/to/u-boot + $ sudo apt-get install python3 python3-virtualenv + $ virtualenv -p /usr/bin/python3 venv + $ . ./venv/bin/activate + $ pip install -r test/py/requirements.txt + +Testing sandbox +--------------- + +To run the test suite on the sandbox port (U-Boot built as a native user-space +application), simply execute: + +.. code-block:: bash + + ./test/py/test.py --bd sandbox --build + +The `--bd` option tells the test suite which board type is being tested. This +lets the test suite know which features the board has, and hence exactly what +can be tested. + +The `--build` option tells U-Boot to compile U-Boot. Alternatively, you may +omit this option and build U-Boot yourself, in whatever way you choose, before +running the test script. + +The test script will attach to U-Boot, execute all valid tests for the board, +then print a summary of the test process. A complete log of the test session +will be written to `${build_dir}/test-log.html`. This is best viewed in a web +browser, but may be read directly as plain text, perhaps with the aid of the +`html2text` utility. + +Testing under a debugger +~~~~~~~~~~~~~~~~~~~~~~~~ + +If you need to run sandbox under a debugger, you may pass the command-line +option `--gdbserver COMM`. This causes two things to happens: + +- Instead of running U-Boot directly, it will be run under gdbserver, with + debug communication via the channel `COMM`. You can attach a debugger to the + sandbox process in order to debug it. See `man gdbserver` and the example + below for details of valid values for `COMM`. +- All timeouts in tests are disabled, allowing U-Boot an arbitrary amount of + time to execute commands. This is useful if U-Boot is stopped at a breakpoint + during debugging. + +A usage example is: + +Window 1: + +.. code-block:: bash + + ./test/py/test.py --bd sandbox --gdbserver localhost:1234 + +Window 2: + +.. code-block:: bash + + gdb ./build-sandbox/u-boot -ex 'target remote localhost:1234' + +Alternatively, you could leave off the `-ex` option and type the command +manually into gdb once it starts. + +You can use any debugger you wish, as long as it speaks the gdb remote +protocol, or any graphical wrapper around gdb. + +Some tests deliberately cause the sandbox process to exit, e.g. to test the +reset command, or sandbox's CTRL-C handling. When this happens, you will need +to attach the debugger to the new sandbox instance. If these tests are not +relevant to your debugging session, you can skip them using pytest's -k +command-line option; see the next section. + +Command-line options +-------------------- + +--board-type, --bd, -B + set the type of the board to be tested. For example, `sandbox` or `seaboard`. + +--board-identity`, --id + sets the identity of the board to be tested. This allows differentiation + between multiple instances of the same type of physical board that are + attached to the same host machine. This parameter is not interpreted by th + test script in any way, but rather is simply passed to the hook scripts + described below, and may be used in any site-specific way deemed necessary. + +--build + indicates that the test script should compile U-Boot itself before running + the tests. If using this option, make sure that any environment variables + required by the build process are already set, such as `$CROSS_COMPILE`. + +--buildman + indicates that `--build` should use buildman to build U-Boot. There is no need + to set $CROSS_COMPILE` in this case since buildman handles it. + +--build-dir + sets the directory containing the compiled U-Boot binaries. If omitted, this + is `${source_dir}/build-${board_type}`. + +--result-dir + sets the directory to write results, such as log files, into. + If omitted, the build directory is used. + +--persistent-data-dir + sets the directory used to store persistent test data. This is test data that + may be re-used across test runs, such as file-system images. + +`pytest` also implements a number of its own command-line options. Commonly used +options are mentioned below. Please see `pytest` documentation for complete +details. Execute `py.test --version` for a brief summary. Note that U-Boot's +test.py script passes all command-line arguments directly to `pytest` for +processing. + +-k + selects which tests to run. The default is to run all known tests. This + option takes a single argument which is used to filter test names. Simple + logical operators are supported. For example: + + - `'-k ums'` runs only tests with "ums" in their name. + - `'-k ut_dm'` runs only tests with "ut_dm" in their name. Note that in this + case, "ut_dm" is a parameter to a test rather than the test name. The full + test name is e.g. "test_ut[ut_dm_leak]". + - `'-k not reset'` runs everything except tests with "reset" in their name. + - `'-k ut or hush'` runs only tests with "ut" or "hush" in their name. + - `'-k not (ut or hush)'` runs everything except tests with "ut" or "hush" in + their name. + +-s + prevents pytest from hiding a test's stdout. This allows you to see + U-Boot's console log in real time on pytest's stdout. + +Testing real hardware +--------------------- + +The tools and techniques used to interact with real hardware will vary +radically between different host and target systems, and the whims of the user. +For this reason, the test suite does not attempt to directly interact with real +hardware in any way. Rather, it executes a standardized set of "hook" scripts +via `$PATH`. These scripts implement certain actions on behalf of the test +suite. This keeps the test suite simple and isolated from system variances +unrelated to U-Boot features. + +Hook scripts +~~~~~~~~~~~~ + +Environment variables +''''''''''''''''''''' + +The following environment variables are set when running hook scripts: + +- `UBOOT_BOARD_TYPE` the board type being tested. +- `UBOOT_BOARD_IDENTITY` the board identity being tested, or `na` if none was + specified. +- `UBOOT_SOURCE_DIR` the U-Boot source directory. +- `UBOOT_TEST_PY_DIR` the full path to `test/py/` in the source directory. +- `UBOOT_BUILD_DIR` the U-Boot build directory. +- `UBOOT_RESULT_DIR` the test result directory. +- `UBOOT_PERSISTENT_DATA_DIR` the test persistent data directory. + +u-boot-test-console +''''''''''''''''''' + +This script provides access to the U-Boot console. The script's stdin/stdout +should be connected to the board's console. This process should continue to run +indefinitely, until killed. The test suite will run this script in parallel +with all other hooks. + +This script may be implemented e.g. by executing `cu`, `kermit`, `conmux`, etc. +via exec(). + +If you are able to run U-Boot under a hardware simulator such as QEMU, then +you would likely spawn that simulator from this script. However, note that +`u-boot-test-reset` may be called multiple times per test script run, and must +cause U-Boot to start execution from scratch each time. Hopefully your +simulator includes a virtual reset button! If not, you can launch the +simulator from `u-boot-test-reset` instead, while arranging for this console +process to always communicate with the current simulator instance. + +u-boot-test-flash +''''''''''''''''' + +Prior to running the test suite against a board, some arrangement must be made +so that the board executes the particular U-Boot binary to be tested. Often +this involves writing the U-Boot binary to the board's flash ROM. The test +suite calls this hook script for that purpose. + +This script should perform the entire flashing process synchronously; the +script should only exit once flashing is complete, and a board reset will +cause the newly flashed U-Boot binary to be executed. + +It is conceivable that this script will do nothing. This might be useful in +the following cases: + +- Some other process has already written the desired U-Boot binary into the + board's flash prior to running the test suite. +- The board allows U-Boot to be downloaded directly into RAM, and executed + from there. Use of this feature will reduce wear on the board's flash, so + may be preferable if available, and if cold boot testing of U-Boot is not + required. If this feature is used, the `u-boot-test-reset` script should + perform this download, since the board could conceivably be reset multiple + times in a single test run. + +It is up to the user to determine if those situations exist, and to code this +hook script appropriately. + +This script will typically be implemented by calling out to some SoC- or +board-specific vendor flashing utility. + +u-boot-test-reset +''''''''''''''''' + +Whenever the test suite needs to reset the target board, this script is +executed. This is guaranteed to happen at least once, prior to executing the +first test function. If any test fails, the test infra-structure will execute +this script again to restore U-Boot to an operational state before running the +next test function. + +This script will likely be implemented by communicating with some form of +relay or electronic switch attached to the board's reset signal. + +The semantics of this script require that when it is executed, U-Boot will +start running from scratch. If the U-Boot binary to be tested has been written +to flash, pulsing the board's reset signal is likely all this script needs to +do. However, in some scenarios, this script may perform other actions. For +example, it may call out to some SoC- or board-specific vendor utility in order +to download the U-Boot binary directly into RAM and execute it. This would +avoid the need for `u-boot-test-flash` to actually write U-Boot to flash, thus +saving wear on the flash chip(s). + +Examples +'''''''' + +https://github.com/swarren/uboot-test-hooks contains some working example hook +scripts, and may be useful as a reference when implementing hook scripts for +your platform. These scripts are not considered part of U-Boot itself. + +Board-type-specific configuration +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +Each board has a different configuration and behaviour. Many of these +differences can be automatically detected by parsing the `.config` file in the +build directory. However, some differences can't yet be handled automatically. + +For each board, an optional Python module `u_boot_board_${board_type}` may exist +to provide board-specific information to the test script. Any global value +defined in these modules is available for use by any test function. The data +contained in these scripts must be purely derived from U-Boot source code. +Hence, these configuration files are part of the U-Boot source tree too. + +Execution environment configuration +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +Each user's hardware setup may enable testing different subsets of the features +implemented by a particular board's configuration of U-Boot. For example, a +U-Boot configuration may support USB device mode and USB Mass Storage, but this +can only be tested if a USB cable is connected between the board and the host +machine running the test script. + +For each board, optional Python modules `u_boot_boardenv_${board_type}` and +`u_boot_boardenv_${board_type}_${board_identity}` may exist to provide +board-specific and board-identity-specific information to the test script. Any +global value defined in these modules is available for use by any test +function. The data contained in these is specific to a particular user's +hardware configuration. Hence, these configuration files are not part of the +U-Boot source tree, and should be installed outside of the source tree. Users +should set `$PYTHONPATH` prior to running the test script to allow these +modules to be loaded. + +Board module parameter usage +~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +The test scripts rely on the following variables being defined by the board +module: + +- none at present + +U-Boot `.config` feature usage +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +The test scripts rely on various U-Boot `.config` features, either directly in +order to test those features, or indirectly in order to query information from +the running U-Boot instance in order to test other features. + +One example is that testing of the `md` command requires knowledge of a RAM +address to use for the test. This data is parsed from the output of the +`bdinfo` command, and hence relies on CONFIG_CMD_BDI being enabled. + +For a complete list of dependencies, please search the test scripts for +instances of: + +- `buildconfig.get(...` +- `@pytest.mark.buildconfigspec(...` +- `@pytest.mark.notbuildconfigspec(...` + +Complete invocation example +~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +Assuming that you have installed the hook scripts into $HOME/ubtest/bin, and +any required environment configuration Python modules into $HOME/ubtest/py, +then you would likely invoke the test script as follows: + +If U-Boot has already been built: + +.. code-block:: bash + + PATH=$HOME/ubtest/bin:$PATH \ + PYTHONPATH=${HOME}/ubtest/py/${HOSTNAME}:${PYTHONPATH} \ + ./test/py/test.py --bd seaboard + +If you want the test script to compile U-Boot for you too, then you likely +need to set `$CROSS_COMPILE` to allow this, and invoke the test script as +follows: + +.. code-block:: bash + + CROSS_COMPILE=arm-none-eabi- \ + PATH=$HOME/ubtest/bin:$PATH \ + PYTHONPATH=${HOME}/ubtest/py/${HOSTNAME}:${PYTHONPATH} \ + ./test/py/test.py --bd seaboard --build + +or, using buildman to handle it: + +.. code-block:: bash + + PATH=$HOME/ubtest/bin:$PATH \ + PYTHONPATH=${HOME}/ubtest/py/${HOSTNAME}:${PYTHONPATH} \ + ./test/py/test.py --bd seaboard --build --buildman + +Writing tests +------------- + +Please refer to the pytest documentation for details of writing pytest tests. +Details specific to the U-Boot test suite are described below. + +A test fixture named `u_boot_console` should be used by each test function. This +provides the means to interact with the U-Boot console, and retrieve board and +environment configuration information. + +The function `u_boot_console.run_command()` executes a shell command on the +U-Boot console, and returns all output from that command. This allows +validation or interpretation of the command output. This function validates +that certain strings are not seen on the U-Boot console. These include shell +error messages and the U-Boot sign-on message (in order to detect unexpected +board resets). See the source of `u_boot_console_base.py` for a complete list of +"bad" strings. Some test scenarios are expected to trigger these strings. Use +`u_boot_console.disable_check()` to temporarily disable checking for specific +strings. See `test_unknown_cmd.py` for an example. + +Board- and board-environment configuration values may be accessed as sub-fields +of the `u_boot_console.config` object, for example +`u_boot_console.config.ram_base`. + +Build configuration values (from `.config`) may be accessed via the dictionary +`u_boot_console.config.buildconfig`, with keys equal to the Kconfig variable +names. diff --git a/test/py/README.md b/test/py/README.md deleted file mode 100644 index fddc104b266..00000000000 --- a/test/py/README.md +++ /dev/null @@ -1,389 +0,0 @@ -# U-Boot pytest suite - -## Introduction - -This tool aims to test U-Boot by executing U-Boot shell commands using the -console interface. A single top-level script exists to execute or attach to the -U-Boot console, run the entire script of tests against it, and summarize the -results. Advantages of this approach are: - -- Testing is performed in the same way a user or script would interact with - U-Boot; there can be no disconnect. -- There is no need to write or embed test-related code into U-Boot itself. - It is asserted that writing test-related code in Python is simpler and more - flexible than writing it all in C. -- It is reasonably simple to interact with U-Boot in this way. - -## Requirements - -The test suite is implemented using pytest. Interaction with the U-Boot console -involves executing some binary and interacting with its stdin/stdout. You will -need to implement various "hook" scripts that are called by the test suite at -the appropriate time. - -In order to run the testsuite at a minimum we require that both python3 and -pip for python3 be installed. All of the required python modules are -described in the requirements.txt file in this directory and can be installed -with the command ```pip install -r requirements.txt``` - -In order to execute certain tests on their supported platforms other tools -will be required. The following is an incomplete list: - -| Package | -| -------------- | -| gdisk | -| dfu-util | -| dtc | -| openssl | -| sudo OR guestmount | -| e2fsprogs | -| util-linux | -| coreutils | -| dosfstools | -| efitools | -| mount | -| mtools | -| sbsigntool | -| udisks2 | - - -Please use the apporirate commands for your distribution to match these tools -up with the package that provides them. - -The test script supports either: - -- Executing a sandbox port of U-Boot on the local machine as a sub-process, - and interacting with it over stdin/stdout. -- Executing an external "hook" scripts to flash a U-Boot binary onto a - physical board, attach to the board's console stream, and reset the board. - Further details are described later. - -### Using `virtualenv` to provide requirements - -The recommended way to run the test suite, in order to ensure reproducibility -is to use `virtualenv` to set up the necessary environment. This can be done -via the following commands: - -```bash -$ cd /path/to/u-boot -$ sudo apt-get install python3 python3-virtualenv -$ virtualenv -p /usr/bin/python3 venv -$ . ./venv/bin/activate -$ pip install -r test/py/requirements.txt -``` - -## Testing sandbox - -To run the testsuite on the sandbox port (U-Boot built as a native user-space -application), simply execute: - -``` -./test/py/test.py --bd sandbox --build -``` - -The `--bd` option tells the test suite which board type is being tested. This -lets the test suite know which features the board has, and hence exactly what -can be tested. - -The `--build` option tells U-Boot to compile U-Boot. Alternatively, you may -omit this option and build U-Boot yourself, in whatever way you choose, before -running the test script. - -The test script will attach to U-Boot, execute all valid tests for the board, -then print a summary of the test process. A complete log of the test session -will be written to `${build_dir}/test-log.html`. This is best viewed in a web -browser, but may be read directly as plain text, perhaps with the aid of the -`html2text` utility. - -### Testing under a debugger - -If you need to run sandbox under a debugger, you may pass the command-line -option `--gdbserver COMM`. This causes two things to happens: - -- Instead of running U-Boot directly, it will be run under gdbserver, with - debug communication via the channel `COMM`. You can attach a debugger to the - sandbox process in order to debug it. See `man gdbserver` and the example - below for details of valid values for `COMM`. -- All timeouts in tests are disabled, allowing U-Boot an arbitrary amount of - time to execute commands. This is useful if U-Boot is stopped at a breakpoint - during debugging. - -A usage example is: - -Window 1: -```shell -./test/py/test.py --bd sandbox --gdbserver localhost:1234 -``` - -Window 2: -```shell -gdb ./build-sandbox/u-boot -ex 'target remote localhost:1234' -``` - -Alternatively, you could leave off the `-ex` option and type the command -manually into gdb once it starts. - -You can use any debugger you wish, so long as it speaks the gdb remote -protocol, or any graphical wrapper around gdb. - -Some tests deliberately cause the sandbox process to exit, e.g. to test the -reset command, or sandbox's CTRL-C handling. When this happens, you will need -to attach the debugger to the new sandbox instance. If these tests are not -relevant to your debugging session, you can skip them using pytest's -k -command-line option; see the next section. - -## Command-line options - -- `--board-type`, `--bd`, `-B` set the type of the board to be tested. For - example, `sandbox` or `seaboard`. -- `--board-identity`, `--id` set the identity of the board to be tested. - This allows differentiation between multiple instances of the same type of - physical board that are attached to the same host machine. This parameter is - not interpreted by the test script in any way, but rather is simply passed - to the hook scripts described below, and may be used in any site-specific - way deemed necessary. -- `--build` indicates that the test script should compile U-Boot itself - before running the tests. If using this option, make sure that any - environment variables required by the build process are already set, such as - `$CROSS_COMPILE`. -- `--buildman` indicates that `--build` should use buildman to build U-Boot. - There is no need to set $CROSS_COMPILE` in this case since buildman handles - it. -- `--build-dir` sets the directory containing the compiled U-Boot binaries. - If omitted, this is `${source_dir}/build-${board_type}`. -- `--result-dir` sets the directory to write results, such as log files, - into. If omitted, the build directory is used. -- `--persistent-data-dir` sets the directory used to store persistent test - data. This is test data that may be re-used across test runs, such as file- - system images. - -`pytest` also implements a number of its own command-line options. Commonly used -options are mentioned below. Please see `pytest` documentation for complete -details. Execute `py.test --version` for a brief summary. Note that U-Boot's -test.py script passes all command-line arguments directly to `pytest` for -processing. - -- `-k` selects which tests to run. The default is to run all known tests. This - option takes a single argument which is used to filter test names. Simple - logical operators are supported. For example: - - `'ums'` runs only tests with "ums" in their name. - - `'ut_dm'` runs only tests with "ut_dm" in their name. Note that in this - case, "ut_dm" is a parameter to a test rather than the test name. The full - test name is e.g. "test_ut[ut_dm_leak]". - - `'not reset'` runs everything except tests with "reset" in their name. - - `'ut or hush'` runs only tests with "ut" or "hush" in their name. - - `'not (ut or hush)'` runs everything except tests with "ut" or "hush" in - their name. -- `-s` prevents pytest from hiding a test's stdout. This allows you to see - U-Boot's console log in real time on pytest's stdout. - -## Testing real hardware - -The tools and techniques used to interact with real hardware will vary -radically between different host and target systems, and the whims of the user. -For this reason, the test suite does not attempt to directly interact with real -hardware in any way. Rather, it executes a standardized set of "hook" scripts -via `$PATH`. These scripts implement certain actions on behalf of the test -suite. This keeps the test suite simple and isolated from system variances -unrelated to U-Boot features. - -### Hook scripts - -#### Environment variables - -The following environment variables are set when running hook scripts: - -- `UBOOT_BOARD_TYPE` the board type being tested. -- `UBOOT_BOARD_IDENTITY` the board identity being tested, or `na` if none was - specified. -- `UBOOT_SOURCE_DIR` the U-Boot source directory. -- `UBOOT_TEST_PY_DIR` the full path to `test/py/` in the source directory. -- `UBOOT_BUILD_DIR` the U-Boot build directory. -- `UBOOT_RESULT_DIR` the test result directory. -- `UBOOT_PERSISTENT_DATA_DIR` the test persistent data directory. - -#### `u-boot-test-console` - -This script provides access to the U-Boot console. The script's stdin/stdout -should be connected to the board's console. This process should continue to run -indefinitely, until killed. The test suite will run this script in parallel -with all other hooks. - -This script may be implemented e.g. by exec()ing `cu`, `kermit`, `conmux`, etc. - -If you are able to run U-Boot under a hardware simulator such as qemu, then -you would likely spawn that simulator from this script. However, note that -`u-boot-test-reset` may be called multiple times per test script run, and must -cause U-Boot to start execution from scratch each time. Hopefully your -simulator includes a virtual reset button! If not, you can launch the -simulator from `u-boot-test-reset` instead, while arranging for this console -process to always communicate with the current simulator instance. - -#### `u-boot-test-flash` - -Prior to running the test suite against a board, some arrangement must be made -so that the board executes the particular U-Boot binary to be tested. Often, -this involves writing the U-Boot binary to the board's flash ROM. The test -suite calls this hook script for that purpose. - -This script should perform the entire flashing process synchronously; the -script should only exit once flashing is complete, and a board reset will -cause the newly flashed U-Boot binary to be executed. - -It is conceivable that this script will do nothing. This might be useful in -the following cases: - -- Some other process has already written the desired U-Boot binary into the - board's flash prior to running the test suite. -- The board allows U-Boot to be downloaded directly into RAM, and executed - from there. Use of this feature will reduce wear on the board's flash, so - may be preferable if available, and if cold boot testing of U-Boot is not - required. If this feature is used, the `u-boot-test-reset` script should - perform this download, since the board could conceivably be reset multiple - times in a single test run. - -It is up to the user to determine if those situations exist, and to code this -hook script appropriately. - -This script will typically be implemented by calling out to some SoC- or -board-specific vendor flashing utility. - -#### `u-boot-test-reset` - -Whenever the test suite needs to reset the target board, this script is -executed. This is guaranteed to happen at least once, prior to executing the -first test function. If any test fails, the test infra-structure will execute -this script again to restore U-Boot to an operational state before running the -next test function. - -This script will likely be implemented by communicating with some form of -relay or electronic switch attached to the board's reset signal. - -The semantics of this script require that when it is executed, U-Boot will -start running from scratch. If the U-Boot binary to be tested has been written -to flash, pulsing the board's reset signal is likely all this script need do. -However, in some scenarios, this script may perform other actions. For -example, it may call out to some SoC- or board-specific vendor utility in order -to download the U-Boot binary directly into RAM and execute it. This would -avoid the need for `u-boot-test-flash` to actually write U-Boot to flash, thus -saving wear on the flash chip(s). - -#### Examples - -https://github.com/swarren/uboot-test-hooks contains some working example hook -scripts, and may be useful as a reference when implementing hook scripts for -your platform. These scripts are not considered part of U-Boot itself. - -### Board-type-specific configuration - -Each board has a different configuration and behaviour. Many of these -differences can be automatically detected by parsing the `.config` file in the -build directory. However, some differences can't yet be handled automatically. - -For each board, an optional Python module `u_boot_board_${board_type}` may exist -to provide board-specific information to the test script. Any global value -defined in these modules is available for use by any test function. The data -contained in these scripts must be purely derived from U-Boot source code. -Hence, these configuration files are part of the U-Boot source tree too. - -### Execution environment configuration - -Each user's hardware setup may enable testing different subsets of the features -implemented by a particular board's configuration of U-Boot. For example, a -U-Boot configuration may support USB device mode and USB Mass Storage, but this -can only be tested if a USB cable is connected between the board and the host -machine running the test script. - -For each board, optional Python modules `u_boot_boardenv_${board_type}` and -`u_boot_boardenv_${board_type}_${board_identity}` may exist to provide -board-specific and board-identity-specific information to the test script. Any -global value defined in these modules is available for use by any test -function. The data contained in these is specific to a particular user's -hardware configuration. Hence, these configuration files are not part of the -U-Boot source tree, and should be installed outside of the source tree. Users -should set `$PYTHONPATH` prior to running the test script to allow these -modules to be loaded. - -### Board module parameter usage - -The test scripts rely on the following variables being defined by the board -module: - -- None at present. - -### U-Boot `.config` feature usage - -The test scripts rely on various U-Boot `.config` features, either directly in -order to test those features, or indirectly in order to query information from -the running U-Boot instance in order to test other features. - -One example is that testing of the `md` command requires knowledge of a RAM -address to use for the test. This data is parsed from the output of the -`bdinfo` command, and hence relies on CONFIG_CMD_BDI being enabled. - -For a complete list of dependencies, please search the test scripts for -instances of: - -- `buildconfig.get(...` -- `@pytest.mark.buildconfigspec(...` -- `@pytest.mark.notbuildconfigspec(...` - -### Complete invocation example - -Assuming that you have installed the hook scripts into $HOME/ubtest/bin, and -any required environment configuration Python modules into $HOME/ubtest/py, -then you would likely invoke the test script as follows: - -If U-Boot has already been built: - -```bash -PATH=$HOME/ubtest/bin:$PATH \ - PYTHONPATH=${HOME}/ubtest/py/${HOSTNAME}:${PYTHONPATH} \ - ./test/py/test.py --bd seaboard -``` - -If you want the test script to compile U-Boot for you too, then you likely -need to set `$CROSS_COMPILE` to allow this, and invoke the test script as -follows: - -```bash -CROSS_COMPILE=arm-none-eabi- \ - PATH=$HOME/ubtest/bin:$PATH \ - PYTHONPATH=${HOME}/ubtest/py/${HOSTNAME}:${PYTHONPATH} \ - ./test/py/test.py --bd seaboard --build -``` - -or, using buildman to handle it: - -```bash - PATH=$HOME/ubtest/bin:$PATH \ - PYTHONPATH=${HOME}/ubtest/py/${HOSTNAME}:${PYTHONPATH} \ - ./test/py/test.py --bd seaboard --build --buildman -``` - -## Writing tests - -Please refer to the pytest documentation for details of writing pytest tests. -Details specific to the U-Boot test suite are described below. - -A test fixture named `u_boot_console` should be used by each test function. This -provides the means to interact with the U-Boot console, and retrieve board and -environment configuration information. - -The function `u_boot_console.run_command()` executes a shell command on the -U-Boot console, and returns all output from that command. This allows -validation or interpretation of the command output. This function validates -that certain strings are not seen on the U-Boot console. These include shell -error messages and the U-Boot sign-on message (in order to detect unexpected -board resets). See the source of `u_boot_console_base.py` for a complete list of -"bad" strings. Some test scenarios are expected to trigger these strings. Use -`u_boot_console.disable_check()` to temporarily disable checking for specific -strings. See `test_unknown_cmd.py` for an example. - -Board- and board-environment configuration values may be accessed as sub-fields -of the `u_boot_console.config` object, for example -`u_boot_console.config.ram_base`. - -Build configuration values (from `.config`) may be accessed via the dictionary -`u_boot_console.config.buildconfig`, with keys equal to the Kconfig variable -names. -- cgit v1.3.1 From 12316967ad5355798a85739a27afbdde35e3c042 Mon Sep 17 00:00:00 2001 From: Heinrich Schuchardt Date: Mon, 18 Jan 2021 21:06:03 +0100 Subject: doc: move README.menu to HTML documentation Convert README.menu to reStructured text and add it to the HTML documentation. Signed-off-by: Heinrich Schuchardt --- doc/README.menu | 124 ----------------------------------------------- doc/develop/index.rst | 1 + doc/develop/menus.rst | 131 ++++++++++++++++++++++++++++++++++++++++++++++++++ 3 files changed, 132 insertions(+), 124 deletions(-) delete mode 100644 doc/README.menu create mode 100644 doc/develop/menus.rst diff --git a/doc/README.menu b/doc/README.menu deleted file mode 100644 index 0f3d7416055..00000000000 --- a/doc/README.menu +++ /dev/null @@ -1,124 +0,0 @@ -SPDX-License-Identifier: GPL-2.0+ -/* - * Copyright 2010-2011 Calxeda, Inc. - */ - -U-Boot provides a set of interfaces for creating and using simple, text -based menus. Menus are displayed as lists of labeled entries on the -console, and an entry can be selected by entering its label. - -To use the menu code, enable CONFIG_MENU, and include "menu.h" where -the interfaces should be available. - -Menus are composed of items. Each item has a key used to identify it in -the menu, and an opaque pointer to data controlled by the consumer. - -If you want to show a menu, instead starting the shell, define -CONFIG_AUTOBOOT_MENU_SHOW. You have to code the int menu_show(int bootdelay) -function, which handle your menu. This function returns the remaining -bootdelay. - -Interfaces ----------- -#include "menu.h" - -/* - * Consumers of the menu interfaces will use a struct menu * as the - * handle for a menu. struct menu is only fully defined in menu.c, - * preventing consumers of the menu interfaces from accessing its - * contents directly. - */ -struct menu; - -/* - * NOTE: See comments in common/menu.c for more detailed documentation on - * these interfaces. - */ - -/* - * menu_create() - Creates a menu handle with default settings - */ -struct menu *menu_create(char *title, int timeout, int prompt, - void (*item_data_print)(void *), - char *(*item_choice)(void *), - void *item_choice_data); - -/* - * menu_item_add() - Adds or replaces a menu item - */ -int menu_item_add(struct menu *m, char *item_key, void *item_data); - -/* - * menu_default_set() - Sets the default choice for the menu - */ -int menu_default_set(struct menu *m, char *item_key); - -/* - * menu_default_choice() - Set *choice to point to the default item's data - */ -int menu_default_choice(struct menu *m, void **choice); - -/* - * menu_get_choice() - Returns the user's selected menu entry, or the - * default if the menu is set to not prompt or the timeout expires. - */ -int menu_get_choice(struct menu *m, void **choice); - -/* - * menu_destroy() - frees the memory used by a menu and its items. - */ -int menu_destroy(struct menu *m); - -/* - * menu_display_statusline(struct menu *m); - * shows a statusline for every menu_display call. - */ -void menu_display_statusline(struct menu *m); - -Example Code ------------- -This example creates a menu that always prompts, and allows the user -to pick from a list of tools. The item key and data are the same. - -#include "menu.h" - -char *tools[] = { - "Hammer", - "Screwdriver", - "Nail gun", - NULL -}; - -char *pick_a_tool(void) -{ - struct menu *m; - int i; - char *tool = NULL; - - m = menu_create("Tools", 0, 1, NULL); - - for(i = 0; tools[i]; i++) { - if (menu_item_add(m, tools[i], tools[i]) != 1) { - printf("failed to add item!"); - menu_destroy(m); - return NULL; - } - } - - if (menu_get_choice(m, (void **)&tool) != 1) - printf("Problem picking tool!\n"); - - menu_destroy(m); - - return tool; -} - -void caller(void) -{ - char *tool = pick_a_tool(); - - if (tool) { - printf("picked a tool: %s\n", tool); - use_tool(tool); - } -} diff --git a/doc/develop/index.rst b/doc/develop/index.rst index 83eb1307edb..beaa64d8d90 100644 --- a/doc/develop/index.rst +++ b/doc/develop/index.rst @@ -12,6 +12,7 @@ Implementation commands global_data logging + menus Debugging --------- diff --git a/doc/develop/menus.rst b/doc/develop/menus.rst new file mode 100644 index 00000000000..dda8f963fb5 --- /dev/null +++ b/doc/develop/menus.rst @@ -0,0 +1,131 @@ +.. SPDX-License-Identifier: GPL-2.0+ +.. Copyright 2010-2011 Calxeda, Inc. + +Menus +===== + +U-Boot provides a set of interfaces for creating and using simple, text +based menus. Menus are displayed as lists of labeled entries on the +console, and an entry can be selected by entering its label. + +To use the menu code, enable CONFIG_MENU, and include "menu.h" where +the interfaces should be available. + +Menus are composed of items. Each item has a key used to identify it in +the menu, and an opaque pointer to data controlled by the consumer. + +If you want to show a menu, instead starting the shell, define +CONFIG_AUTOBOOT_MENU_SHOW. You have to code the int menu_show(int bootdelay) +function, which handle your menu. This function returns the remaining +bootdelay. + +Interfaces +---------- + +.. code-block:: c + + #include "menu.h" + + /* + * Consumers of the menu interfaces will use a struct menu * as the + * handle for a menu. struct menu is only fully defined in menu.c, + * preventing consumers of the menu interfaces from accessing its + * contents directly. + */ + struct menu; + + /* + * NOTE: See comments in common/menu.c for more detailed documentation on + * these interfaces. + */ + + /* + * menu_create() - Creates a menu handle with default settings + */ + struct menu *menu_create(char *title, int timeout, int prompt, + void (*item_data_print)(void *), + char *(*item_choice)(void *), + void *item_choice_data); + + /* + * menu_item_add() - Adds or replaces a menu item + */ + int menu_item_add(struct menu *m, char *item_key, void *item_data); + + /* + * menu_default_set() - Sets the default choice for the menu + */ + int menu_default_set(struct menu *m, char *item_key); + + /* + * menu_default_choice() - Set *choice to point to the default item's data + */ + int menu_default_choice(struct menu *m, void **choice); + + /* + * menu_get_choice() - Returns the user's selected menu entry, or the + * default if the menu is set to not prompt or the timeout expires. + */ + int menu_get_choice(struct menu *m, void **choice); + + /* + * menu_destroy() - frees the memory used by a menu and its items. + */ + int menu_destroy(struct menu *m); + + /* + * menu_display_statusline(struct menu *m); + * shows a statusline for every menu_display call. + */ + void menu_display_statusline(struct menu *m); + +Example Code +------------ + +This example creates a menu that always prompts, and allows the user +to pick from a list of tools. The item key and data are the same. + +.. code-block:: c + + #include "menu.h" + + char *tools[] = { + "Hammer", + "Screwdriver", + "Nail gun", + NULL + }; + + char *pick_a_tool(void) + { + struct menu *m; + int i; + char *tool = NULL; + + m = menu_create("Tools", 0, 1, NULL); + + for(i = 0; tools[i]; i++) { + if (menu_item_add(m, tools[i], tools[i]) != 1) { + printf("failed to add item!"); + menu_destroy(m); + return NULL; + } + } + + if (menu_get_choice(m, (void **)&tool) != 1) + printf("Problem picking tool!\n"); + + menu_destroy(m); + + return tool; + } + + void caller(void) + { + char *tool = pick_a_tool(); + + if (tool) { + printf("picked a tool: %s\n", tool); + use_tool(tool); + } + } -- cgit v1.3.1