From c30bdf7fcad5ef7a0092d779abad4ee6fc331470 Mon Sep 17 00:00:00 2001 From: John Fairhurst Date: Sat, 7 Nov 2020 10:33:08 +0000 Subject: [PATCH] Swift SymbolGraph importer (#1171) Add `--swift-build-tool symbolgraph` to use the Swift 5.3 symbolgraph feature to generate docs from `.swiftmodule` files. Add to test specs and docs. The importer converts the symbol graph to approximately SourceKitten output and provides that as input to the existing code. Only intentional change to existing code is a modification of the "do we blame the user for this declaration not being documented" question, to deal with symbol graph output that is sometimes missing source locations. The modification turned out to fix #498 and showed a missed 'undocumented' declaration in a test suite. --- CHANGELOG.md | 4 +- README.md | 61 +++++++- lib/jazzy/config.rb | 10 +- lib/jazzy/doc_builder.rb | 8 +- lib/jazzy/sourcekitten.rb | 11 +- lib/jazzy/symbol_graph.rb | 95 +++++++++++ lib/jazzy/symbol_graph/constraint.rb | 94 +++++++++++ lib/jazzy/symbol_graph/ext_node.rb | 114 ++++++++++++++ lib/jazzy/symbol_graph/graph.rb | 193 +++++++++++++++++++++++ lib/jazzy/symbol_graph/relationship.rb | 39 +++++ lib/jazzy/symbol_graph/sym_node.rb | 154 ++++++++++++++++++ lib/jazzy/symbol_graph/symbol.rb | 208 +++++++++++++++++++++++++ spec/integration_spec.rb | 13 +- spec/integration_specs | 2 +- 14 files changed, 987 insertions(+), 19 deletions(-) create mode 100644 lib/jazzy/symbol_graph.rb create mode 100644 lib/jazzy/symbol_graph/constraint.rb create mode 100644 lib/jazzy/symbol_graph/ext_node.rb create mode 100644 lib/jazzy/symbol_graph/graph.rb create mode 100644 lib/jazzy/symbol_graph/relationship.rb create mode 100644 lib/jazzy/symbol_graph/sym_node.rb create mode 100644 lib/jazzy/symbol_graph/symbol.rb diff --git a/CHANGELOG.md b/CHANGELOG.md index e81f75ddc..d0165badc 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -6,7 +6,9 @@ ##### Enhancements -* None. +* Support documentation generation from `.swiftmodule` binaries using + `--swift-build-tool symbolgraph` with Swift 5.3. + [John Fairhurst](https://github.com/johnfairh) ##### Bug Fixes diff --git a/README.md b/README.md index 350fe7e95..5ce1ff4ee 100644 --- a/README.md +++ b/README.md @@ -8,13 +8,14 @@ Both Swift and Objective-C projects are supported. -*SwiftPM support was recently added, so please report any issues you find.* - Instead of parsing your source files, `jazzy` hooks into [Clang][clang] and [SourceKit][sourcekit] to use the [AST][ast] representation of your code and its comments for more accurate results. The output matches the look and feel of Apple’s official reference documentation, post WWDC 2014. +Jazzy can also generate documentation from compiled swift modules [using their +symbol graph](#docs-from-swiftmodules-or-frameworks) instead of source code. + ![Screenshot](images/screenshot.jpg) This project adheres to the [Contributor Covenant Code of Conduct](https://realm.io/conduct). @@ -49,13 +50,14 @@ succeed too! If Jazzy generates docs for the wrong module then use `--module` to tell it which one you'd prefer. If this doesn't help, and you're using Xcode, then try passing -extra arguments to `xcodebuild`, for example `jazzy --build-tool-arguments -target,MyTarget`. +extra arguments to `xcodebuild`, for example +`jazzy --build-tool-arguments -scheme,MyScheme,-target,MyTarget`. You can set options for your project’s documentation in a configuration file, `.jazzy.yaml` by default. For a detailed explanation and an exhaustive list of all available options, run `jazzy --help config`. -### Supported keywords +### Supported documentation keywords Swift documentation is written in markdown and supports a number of special keywords. For a complete list and examples, see Erica Sadun's post on [*Swift header documentation in Xcode 7*](https://ericasadun.com/2015/06/14/swift-header-documentation-in-xcode-7/), @@ -200,6 +202,57 @@ sourcekitten doc --objc $(pwd)/MyProject/MyProject.h \ jazzy --sourcekitten-sourcefile swiftDoc.json,objcDoc.json ``` +### Docs from `.swiftmodule`s or frameworks + +*This feature is new and relies on a new Swift feature: there may be crashes +and mistakes: reports welcome.* + +Swift 5.3 adds support for symbol graph generation from `.swiftmodule` files. +This looks to be part of Apple's toolchain for generating their online docs. + +Jazzy can use this to generate API documentation. This is faster than using +the source code directly but does have limitations: for example documentation +comments are available only for `public` declarations, and the presentation of +Swift extensions may not match the way they are written in code. + +Some examples: + +1. Generate docs for the Apple Combine framework for macOS: + ```shell + jazzy --module Combine --swift-build-tool symbolgraph + ``` + The SDK's library directories are included in the search path by + default. +2. Same but for iOS: + ```shell + jazzy --module Combine --swift-build-tool symbolgraph + --sdk iphoneos + --build-tool-arguments --target,arm64-apple-ios14.1 + ``` + The `target` is the LLVM target triple and needs to match the SDK. The + default here is the target of the host system that Jazzy is running on, + something like `x86_64-apple-darwin19.6.0`. +3. Generate docs for a personal `.swiftmodule`: + ```shell + jazzy --module MyMod --swift-build-tool symbolgraph + --build-tool-arguments -I,/Build/Products + ``` + This implies that `/Build/Products/MyMod.swiftmodule` exists. Jazzy's + `--source-directory` (default current directory) is searched by default, + so you only need the `-I` override if that's not enough. +4. For a personal framework: + ```shell + jazzy --module MyMod --swift-build-tool symbolgraph + --build-tool-arguments -F,/Build/Products + ``` + This implies that `/Build/Products/MyMod.framework` exists and contains + a `.swiftmodule`. Again the `--source-directory` is searched by default + if `-F` is not passed in. + +See `swift symbolgraph-extract --help` for all the things you can pass via +`--build-tool-arguments`: if your module has dependencies then you may need +to add various search path options to let Swift load it. + ### Themes Three themes are provided with jazzy: `apple` (default), `fullwidth` and `jony`. diff --git a/lib/jazzy/config.rb b/lib/jazzy/config.rb index b939b6191..4e0819716 100644 --- a/lib/jazzy/config.rb +++ b/lib/jazzy/config.rb @@ -219,14 +219,14 @@ def hide_objc? end end - SWIFT_BUILD_TOOLS = %w[spm xcodebuild].freeze + SWIFT_BUILD_TOOLS = %w[spm xcodebuild symbolgraph].freeze config_attr :swift_build_tool, command_line: "--swift-build-tool #{SWIFT_BUILD_TOOLS.join(' | ')}", - description: 'Control whether Jazzy uses Swift Package Manager or '\ - 'xcodebuild to build the module to be documented. By '\ - 'default it uses xcodebuild if there is a .xcodeproj '\ - 'file in the source directory.', + description: 'Control whether Jazzy uses Swift Package Manager, '\ + 'xcodebuild, or swift-symbolgraph to build the module '\ + 'to be documented. By default it uses xcodebuild if '\ + 'there is a .xcodeproj file in the source directory.', parse: ->(tool) do return tool.to_sym if SWIFT_BUILD_TOOLS.include?(tool) raise "Unsupported swift_build_tool #{tool}, "\ diff --git a/lib/jazzy/doc_builder.rb b/lib/jazzy/doc_builder.rb index 914cfae2a..6a65e2c73 100644 --- a/lib/jazzy/doc_builder.rb +++ b/lib/jazzy/doc_builder.rb @@ -14,6 +14,7 @@ require 'jazzy/source_document' require 'jazzy/source_module' require 'jazzy/sourcekitten' +require 'jazzy/symbol_graph' module Jazzy # This module handles HTML generation, file writing, asset copying, @@ -72,6 +73,8 @@ def self.build(options) elsif options.podspec_configured pod_documenter = PodspecDocumenter.new(options.podspec) stdout = pod_documenter.sourcekitten_output(options) + elsif options.swift_build_tool == :symbolgraph + stdout = SymbolGraph.build(options) else stdout = Dir.chdir(options.source_directory) do arguments = SourceKitten.arguments_from_options(options) @@ -192,7 +195,10 @@ def self.write_lint_report(undocumented, options) lint_report = { warnings: warnings.sort_by do |w| - [w[:file], w[:line] || 0, w[:symbol], w[:symbol_kind]] + [w[:file] || Pathname(''), + w[:line] || 0, + w[:symbol], + w[:symbol_kind]] end, source_directory: options.source_directory, } diff --git a/lib/jazzy/sourcekitten.rb b/lib/jazzy/sourcekitten.rb index 2d6a5ccff..a3ba92434 100644 --- a/lib/jazzy/sourcekitten.rb +++ b/lib/jazzy/sourcekitten.rb @@ -325,17 +325,16 @@ def self.should_document_swift_extension?(doc) end end - def self.should_mark_undocumented(filepath) - source_directory = Config.instance.source_directory.to_s - (filepath || '').start_with?(source_directory) + # Call things undocumented if they were compiled properly + # and came from our module. + def self.should_mark_undocumented(declaration) + declaration.usr && !declaration.modulename end def self.process_undocumented_token(doc, declaration) make_default_doc_info(declaration) - filepath = doc['key.filepath'] - - if !declaration.swift? || should_mark_undocumented(filepath) + if !declaration.swift? || should_mark_undocumented(declaration) @stats.add_undocumented(declaration) return nil if @skip_undocumented declaration.abstract = undocumented_abstract diff --git a/lib/jazzy/symbol_graph.rb b/lib/jazzy/symbol_graph.rb new file mode 100644 index 000000000..216aa200d --- /dev/null +++ b/lib/jazzy/symbol_graph.rb @@ -0,0 +1,95 @@ +require 'set' +require 'jazzy/symbol_graph/graph' +require 'jazzy/symbol_graph/constraint' +require 'jazzy/symbol_graph/symbol' +require 'jazzy/symbol_graph/relationship' +require 'jazzy/symbol_graph/sym_node' +require 'jazzy/symbol_graph/ext_node' + +# This is the top-level symbolgraph driver that deals with +# figuring out arguments, running the tool, and loading the +# results. + +module Jazzy + module SymbolGraph + # Run `swift symbolgraph-extract` with configured args, + # parse the results, and return as JSON in SourceKit[ten] + # format. + def self.build(config) + Dir.mktmpdir do |tmp_dir| + args = arguments(config, tmp_dir) + + Executable.execute_command('swift', + args.unshift('symbolgraph-extract'), + true) # raise on error + + Dir[tmp_dir + '/*.symbols.json'].map do |filename| + # The @ part is for extensions in our module (before the @) + # of types in another module (after the @). + filename =~ /(.*?)(@(.*?))?\.symbols/ + module_name = Regexp.last_match[3] || Regexp.last_match[1] + { + filename => + Graph.new(File.read(filename), module_name).to_sourcekit, + } + end.to_json + end + end + + # Figure out the args to pass to symbolgraph-extract + # rubocop:disable Metrics/CyclomaticComplexity + def self.arguments(config, output_path) + if config.module_name.empty? + raise 'error: `--swift-build-tool symbolgraph` requires `--module`.' + end + + user_args = config.build_tool_arguments.join + + if user_args =~ /--(?:module-name|minimum-access-level|output-dir)/ + raise 'error: `--build-tool-arguments` for '\ + "`--swift-build-tool symbolgraph` can't use `--module`, "\ + '`--minimum-access-level`, or `--output-dir`.' + end + + # Default set + args = [ + "--module-name=#{config.module_name}", + '--minimum-access-level=private', + "--output-dir=#{output_path}", + '--skip-synthesized-members', + ] + + # Things user can override + args.append("--sdk=#{sdk(config)}") unless user_args =~ /--sdk/ + args.append("--target=#{target}") unless user_args =~ /--target/ + args.append("-F=#{config.source_directory}") unless user_args =~ /-F(?!s)/ + args.append("-I=#{config.source_directory}") unless user_args =~ /-I/ + + args + config.build_tool_arguments + end + # rubocop:enable Metrics/CyclomaticComplexity + + # Get the SDK path. On !darwin this just isn't needed. + def self.sdk(config) + `xcrun --show-sdk-path --sdk #{config.sdk}`.chomp + end + + # Guess a default LLVM target. Feels like the tool should figure this + # out from sdk + the binary somehow? + def self.target + `swift -version` =~ /Target: (.*?)$/ + Regexp.last_match[1] || 'x86_64-apple-macosx10.15' + end + + # This is a last-ditch fallback for when symbolgraph doesn't + # provide a name - at least conforming external types to local + # protocols. + def self.demangle(usr) + args = %w[demangle -simplified -compact].append(usr.sub(/^s:/, 's')) + output, = Executable.execute_command('swift', args, true) + return output.chomp + rescue + usr + end + end +end diff --git a/lib/jazzy/symbol_graph/constraint.rb b/lib/jazzy/symbol_graph/constraint.rb new file mode 100644 index 000000000..3127e3518 --- /dev/null +++ b/lib/jazzy/symbol_graph/constraint.rb @@ -0,0 +1,94 @@ +module Jazzy + module SymbolGraph + # Constraint is a tidied-up JSON object, used by both Symbol and + # Relationship, and key to reconstructing extensions. + class Constraint + attr_accessor :kind + attr_accessor :lhs + attr_accessor :rhs + + private + + def initialize(kind, lhs, rhs) + self.kind = kind # "==" or ":" + self.lhs = lhs + self.rhs = rhs + end + + public + + KIND_MAP = { + 'conformance' => ':', + 'superclass' => ':', + 'sameType' => '==', + }.freeze + + # Init from a JSON hash + def self.new_hash(hash) + kind = KIND_MAP[hash[:kind]] + raise "Unknown constraint kind '#{kind}'" unless kind + lhs = hash[:lhs].sub(/^Self\./, '') + rhs = hash[:rhs].sub(/^Self\./, '') + new(kind, lhs, rhs) + end + + # Init from a Swift declaration fragment eg. 'A : B' + def self.new_declaration(decl) + decl =~ /^(.*?)\s*([:<=]+)\s*(.*)$/ + new(Regexp.last_match[2], + Regexp.last_match[1], + Regexp.last_match[3]) + end + + def to_swift + "#{lhs} #{kind} #{rhs}" + end + + # The first component of types in the constraint + def type_names + Set.new([lhs, rhs].map { |n| n.sub(/\..*$/, '') }) + end + + def self.new_list(hash_list) + hash_list.map { |h| Constraint.new_hash(h) }.sort.uniq + end + + # Swift protocols and reqs have an implementation/hidden conformance + # to their own protocol: we don't want to think about this in docs. + def self.new_list_for_symbol(hash_list, path_components) + hash_list.map do |hash| + if hash[:lhs] == 'Self' && + hash[:kind] == 'conformance' && + path_components.include?(hash[:rhs]) + next nil + end + Constraint.new_hash(hash) + end.compact + end + + # Workaround Swift 5.3 bug with missing constraint rels + def self.new_list_from_declaration(decl) + decl.split(/\s*,\s*/).map { |cons| Constraint.new_declaration(cons) } + end + + # Sort order - by Swift text + include Comparable + + def <=>(other) + to_swift <=> other.to_swift + end + + alias eql? == + + def hash + to_swift.hash + end + end + end +end + +class Array + def to_where_clause + empty? ? '' : ' where ' + map(&:to_swift).join(', ') + end +end diff --git a/lib/jazzy/symbol_graph/ext_node.rb b/lib/jazzy/symbol_graph/ext_node.rb new file mode 100644 index 000000000..5581ee5f3 --- /dev/null +++ b/lib/jazzy/symbol_graph/ext_node.rb @@ -0,0 +1,114 @@ +module Jazzy + module SymbolGraph + # For extensions we need to track constraints of the extended type + # and the constraints introduced by the extension. + class ExtConstraints + attr_accessor :type # array + attr_accessor :ext # array + + # all constraints inherited by members of the extension + def merged + (type + ext).sort + end + + def initialize(type_constraints, ext_constraints) + self.type = type_constraints || [] + self.ext = ext_constraints || [] + end + end + + # An ExtNode is a node of the reconstructed syntax tree representing + # an extension that we fabricate to resolve certain relationships. + class ExtNode < BaseNode + attr_accessor :usr + attr_accessor :name + attr_accessor :all_constraints # ExtConstraints + attr_accessor :conformances # array, can be empty + + # Deduce an extension from a member of an unknown type or + # of known type with additional constraints + def self.new_for_member(type_usr, + member, + constraints) + new(type_usr, + member.parent_qualified_name, + constraints).tap { |o| o.add_child(member) } + end + + # Deduce an extension from a protocol conformance for some type + def self.new_for_conformance(type_usr, + type_name, + protocol, + constraints) + new(type_usr, type_name, constraints).tap do |o| + o.add_conformance(protocol) + end + end + + private + + def initialize(usr, name, constraints) + self.usr = usr + self.name = name + self.all_constraints = constraints + self.conformances = [] + super() + end + + public + + def constraints + all_constraints.merged + end + + def add_conformance(protocol) + conformances.append(protocol).sort! + end + + def full_declaration + decl = "extension #{name}" + unless conformances.empty? + decl += ' : ' + conformances.join(', ') + end + decl + all_constraints.ext.to_where_clause + end + + def to_sourcekit(module_name) + declaration = full_declaration + xml_declaration = "#{CGI.escapeHTML(declaration)}" + + hash = { + 'key.kind' => 'source.lang.swift.decl.extension', + 'key.usr' => usr, + 'key.name' => name, + 'key.modulename' => module_name, + 'key.parsed_declaration' => declaration, + 'key.annotated_decl' => xml_declaration, + } + + unless conformances.empty? + hash['key.inheritedtypes'] = conformances.map do |conformance| + { 'key.name' => conformance } + end + end + + unless children.empty? + hash['key.substructure'] = children_to_sourcekit + end + + hash + end + + # Sort order - by type name then constraint + include Comparable + + def sort_key + name + constraints.map(&:to_swift).join + end + + def <=>(other) + sort_key <=> other.sort_key + end + end + end +end diff --git a/lib/jazzy/symbol_graph/graph.rb b/lib/jazzy/symbol_graph/graph.rb new file mode 100644 index 000000000..cdac6fbe8 --- /dev/null +++ b/lib/jazzy/symbol_graph/graph.rb @@ -0,0 +1,193 @@ +# rubocop:disable Metrics/ClassLength +module Jazzy + module SymbolGraph + # A Graph is the coordinator to import a symbolgraph json file. + # Deserialize it to Symbols and Relationships, then rebuild + # the AST shape using SymNodes and ExtNodes and extract SourceKit json. + class Graph + attr_accessor :module_name + attr_accessor :symbol_nodes # usr -> SymNode + attr_accessor :relationships # [Relationship] + attr_accessor :ext_nodes # (usr, constraints) -> ExtNode + + # Parse the JSON into flat tables of data + def initialize(json, module_name) + self.module_name = module_name + graph = JSON.parse(json, symbolize_names: true) + + self.symbol_nodes = {} + graph[:symbols].each do |hash| + symbol = Symbol.new(hash) + symbol_nodes[symbol.usr] = SymNode.new(symbol) + end + + self.relationships = + graph[:relationships].map { |hash| Relationship.new(hash) } + + self.ext_nodes = {} + end + + # ExtNode index. (type USR, extension constraints) -> ExtNode. + # This minimizes the number of extensions + + def ext_key(usr, constraints) + usr + constraints.map(&:to_swift).join + end + + def add_ext_member(type_usr, member_node, constraints) + key = ext_key(type_usr, constraints.ext) + if ext_node = ext_nodes[key] + ext_node.add_child(member_node) + else + ext_nodes[key] = + ExtNode.new_for_member(type_usr, member_node, constraints) + end + end + + def add_ext_conformance(type_usr, + type_name, + protocol, + constraints) + key = ext_key(type_usr, constraints.ext) + if ext_node = ext_nodes[key] + ext_node.add_conformance(protocol) + else + ext_nodes[key] = + ExtNode.new_for_conformance(type_usr, + type_name, + protocol, + constraints) + end + end + + # Increasingly desparate ways to find the name of the symbol + # at the target end of a relationship + def rel_target_name(rel, target_node) + (target_node && target_node.symbol.name) || + rel.target_fallback || + Jazzy::SymbolGraph.demangle(rel.target_usr) + end + + # Same for the source end. Less help from the tool here + def rel_source_name(rel, source_node) + (source_node && source_node.qualified_name) || + Jazzy::SymbolGraph.demangle(rel.source_usr) + end + + # Protocol conformance is redundant if it's unconditional + # and already expressed in the type's declaration. + def redundant_conformance?(rel, type, protocol) + type && rel.constraints.empty? && type.conformance?(protocol) + end + + # source is a member/protocol requirement of target + def rebuild_member(rel, source, target) + return unless source + + source.protocol_requirement = rel.protocol_requirement? + constraints = + ExtConstraints.new(target && target.constraints, + source.unique_context_constraints(target)) + + # Add to its parent or invent an extension + unless target && target.try_add_child(source, constraints.ext) + add_ext_member(rel.target_usr, source, constraints) + end + end + + # "source : target" either from type decl or ext decl + def rebuild_conformance(rel, source, target) + protocol_name = rel_target_name(rel, target) + + return if redundant_conformance?(rel, source, protocol_name) + + type_constraints = (source && source.constraints) || [] + constraints = + ExtConstraints.new(type_constraints, + rel.constraints - type_constraints) + + # Create an extension or enhance an existing one + add_ext_conformance(rel.source_usr, + rel_source_name(rel, source), + protocol_name, + constraints) + end + + # "source is a default implementation of protocol requirement target" + def rebuild_default_implementation(_rel, source, target) + return unless source + + unless target && + (target_parent = target.parent) && + target_parent.is_a?(SymNode) + # Could probably figure this out with demangle, but... + warn "Can't resolve membership of default implementation "\ + "#{source.symbol.usr}." + source.unlisted = true + return + end + constraints = + ExtConstraints.new(target_parent.constraints, + source.unique_context_constraints(target_parent)) + + add_ext_member(target_parent.symbol.usr, + source, + constraints) + end + + # "source is a class that inherits from target" + def rebuild_inherits(_rel, source, target) + if source && target + source.superclass_name = target.symbol.name + end + end + + # Process a structural relationship to link nodes + def rebuild_rel(rel) + source = symbol_nodes[rel.source_usr] + target = symbol_nodes[rel.target_usr] + + case rel.kind + when :memberOf, :optionalRequirementOf, :requirementOf + rebuild_member(rel, source, target) + + when :conformsTo + rebuild_conformance(rel, source, target) + + when :defaultImplementationOf + rebuild_default_implementation(rel, source, target) + + when :inheritsFrom + rebuild_inherits(rel, source, target) + end + # don't seem to care about: + # - overrides: not bothered, also unimplemented for protocols + end + + # Rebuild the AST structure and convert to SourceKit + def to_sourcekit + # Do default impls after the others so we can find protocol + # type nodes from protocol requirements. + default_impls, other_rels = + relationships.partition(&:default_implementation?) + (other_rels + default_impls).each { |r| rebuild_rel(r) } + + root_symbol_nodes = + symbol_nodes.values + .select(&:top_level_decl?) + .sort + .map(&:to_sourcekit) + + root_ext_nodes = + ext_nodes.values + .sort + .map { |n| n.to_sourcekit(module_name) } + { + 'key.diagnostic_stage' => 'parse', + 'key.substructure' => root_symbol_nodes + root_ext_nodes, + } + end + end + end +end +# rubocop:enable Metrics/ClassLength diff --git a/lib/jazzy/symbol_graph/relationship.rb b/lib/jazzy/symbol_graph/relationship.rb new file mode 100644 index 000000000..f59d63942 --- /dev/null +++ b/lib/jazzy/symbol_graph/relationship.rb @@ -0,0 +1,39 @@ +module Jazzy + module SymbolGraph + # A Relationship is a tidied-up SymbolGraph JSON object + class Relationship + attr_accessor :kind + attr_accessor :source_usr + attr_accessor :target_usr + attr_accessor :target_fallback # can be nil + attr_accessor :constraints # array, can be empty + + KINDS = %w[memberOf conformsTo defaultImplementationOf + overrides inheritsFrom requirementOf + optionalRequirementOf].freeze + + def protocol_requirement? + %i[requirementOf optionalRequirementOf].include? kind + end + + def default_implementation? + kind == :defaultImplementationOf + end + + def initialize(hash) + kind = hash[:kind] + unless KINDS.include?(kind) + raise "Unknown relationship kind '#{kind}'" + end + self.kind = kind.to_sym + self.source_usr = hash[:source] + self.target_usr = hash[:target] + if fallback = hash[:targetFallback] + # Strip the leading module name + self.target_fallback = fallback.sub(/^.*?\./, '') + end + self.constraints = Constraint.new_list(hash[:swiftConstraints] || []) + end + end + end +end diff --git a/lib/jazzy/symbol_graph/sym_node.rb b/lib/jazzy/symbol_graph/sym_node.rb new file mode 100644 index 000000000..9b7e98d0e --- /dev/null +++ b/lib/jazzy/symbol_graph/sym_node.rb @@ -0,0 +1,154 @@ +module Jazzy + module SymbolGraph + # The rebuilt syntax tree is made of nodes that either match + # symbols or that we fabricate for extensions. This is the common + # treeishness. + class BaseNode + attr_accessor :children # array, can be empty + attr_accessor :parent # can be nil + + def initialize + self.children = [] + end + + def add_child(child) + child.parent = self + children.append(child) + end + + def children_to_sourcekit + children.sort.map(&:to_sourcekit) + end + end + + # A SymNode is a node of the reconstructed syntax tree holding a symbol. + # It can turn itself into SourceKit and helps decode extensions. + class SymNode < BaseNode + attr_accessor :symbol + attr_writer :override + attr_writer :protocol_requirement + attr_writer :unlisted + attr_accessor :superclass_name + + def override? + @override + end + + def protocol_requirement? + @protocol_requirement + end + + def top_level_decl? + !@unlisted && parent.nil? + end + + def initialize(symbol) + self.symbol = symbol + super() + end + + def qualified_name + symbol.path_components.join('.') + end + + def parent_qualified_name + symbol.path_components[0...-1].join('.') + end + + def protocol? + symbol.kind.end_with?('protocol') + end + + def constraints + symbol.constraints + end + + # Add another SymNode as a member if possible. + # It must go in an extension if either: + # - it has different generic constraints to us; or + # - we're a protocol and it's a default impl / ext method + def try_add_child(node, unique_context_constraints) + unless unique_context_constraints.empty? && + (!protocol? || node.protocol_requirement?) + return false + end + add_child(node) + true + end + + # The `Constraint`s on this decl that are both: + # 1. Unique, ie. not just inherited from its context; and + # 2. Constraining the *context's* gen params rather than our own. + def unique_context_constraints(context) + return symbol.constraints unless context + + new_generic_type_params = + symbol.generic_type_params - context.symbol.generic_type_params + + (symbol.constraints - context.symbol.constraints) + .select { |con| con.type_names.disjoint?(new_generic_type_params) } + end + + # Messy check whether we need to fabricate an extension for a protocol + # conformance: don't bother if it's already in the type declaration. + def conformance?(protocol) + return false unless symbol.declaration =~ /(?<=:).*?(?=(where|$))/ + Regexp.last_match[0] =~ /\b#{protocol}\b/ + end + + # Generate the 'where' clause for the declaration + def where_clause + parent_constraints = (parent && parent.constraints) || [] + (constraints - parent_constraints).to_where_clause + end + + def inherits_clause + return '' unless superclass_name + " : #{superclass_name}" + end + + def full_declaration + symbol.availability + .append(symbol.declaration + inherits_clause + where_clause) + .join("\n") + end + + # rubocop:disable Metrics/MethodLength + def to_sourcekit + declaration = full_declaration + xml_declaration = "#{CGI.escapeHTML(declaration)}" + + hash = { + 'key.kind' => symbol.kind, + 'key.usr' => symbol.usr, + 'key.name' => symbol.name, + 'key.accessibility' => symbol.acl, + 'key.parsed_decl' => declaration, + 'key.annotated_decl' => xml_declaration, + } + if docs = symbol.doc_comments + hash['key.doc.comment'] = docs + hash['key.doc.full_as_xml'] = '' + end + if location = symbol.location + hash['key.filepath'] = location[:filename] + hash['key.doc.line'] = location[:line] + hash['key.doc.column'] = location[:character] + end + unless children.empty? + hash['key.substructure'] = children_to_sourcekit + end + + hash + end + # rubocop:enable Metrics/MethodLength + + # Sort order - by symbol + include Comparable + + def <=>(other) + symbol <=> other.symbol + end + end + end +end diff --git a/lib/jazzy/symbol_graph/symbol.rb b/lib/jazzy/symbol_graph/symbol.rb new file mode 100644 index 000000000..bf1087e00 --- /dev/null +++ b/lib/jazzy/symbol_graph/symbol.rb @@ -0,0 +1,208 @@ +# rubocop:disable Metrics/ClassLength +module Jazzy + module SymbolGraph + # A Symbol is a tidied-up SymbolGraph JSON object + class Symbol + attr_accessor :usr + attr_accessor :path_components + attr_accessor :declaration + attr_accessor :kind + attr_accessor :acl + attr_accessor :location # can be nil, keys :filename :line :character + attr_accessor :constraints # array, can be empty + attr_accessor :doc_comments # can be nil + attr_accessor :availability # array, can be empty + attr_accessor :generic_type_params # set, can be empty + + def name + path_components[-1] || '??' + end + + def initialize(hash) + self.usr = hash[:identifier][:precise] + self.path_components = hash[:pathComponents] + raw_decl = hash[:declarationFragments].map { |f| f[:spelling] }.join + init_kind(hash[:kind][:identifier]) + init_declaration(raw_decl) + init_acl(hash[:accessLevel]) + if location = hash[:location] + init_location(location) + end + init_constraints(hash, raw_decl) + if comments_hash = hash[:docComment] + init_doc_comments(comments_hash) + end + init_availability(hash[:availability] || []) + init_generic_type_params(hash) + end + + # Repair problems with SymbolGraph's declprinter + + def init_declaration(raw_decl) + # Too much 'Self.TypeName'; omitted arg labels look odd; + # duplicated constraints; swift 5.3 vs. master workaround + self.declaration = + raw_decl.gsub(/\bSelf\./, '') + .gsub(/(?<=\(|, )_: /, '_ arg: ') + .gsub(/ where.*$/, '') + if kind == 'source.lang.swift.decl.class' + declaration.sub!(/\s*:.*$/, '') + end + end + + # Mapping SymbolGraph's declkinds to SourceKit + + KIND_MAP = { + 'class' => 'class', + 'struct' => 'struct', + 'enum' => 'enum', + 'enum.case' => 'enumelement', # intentional + 'protocol' => 'protocol', + 'init' => 'function.constructor', + 'deinit' => 'function.destructor', + 'func.op' => 'function.operator', + 'type.method' => 'function.method.class', + 'static.method' => 'function.method.static', + 'method' => 'function.method.instance', + 'func' => 'function.free', + 'type.property' => 'var.class', + 'static.property' => 'var.static', + 'property' => 'var.instance', + 'var' => 'var.global', + 'subscript' => 'function.subscript', + 'type.subscript' => 'function.subscript', + 'static.subscript' => 'function.subscript', + 'typealias' => 'typealias', + 'associatedtype' => 'associatedtype', + }.freeze + + # We treat 'static var' differently to 'class var' + def adjust_kind_for_declaration(kind) + return kind unless declaration =~ /\bstatic\b/ + kind.gsub(/type/, 'static') + end + + def init_kind(kind) + adjusted = adjust_kind_for_declaration(kind) + sourcekit_kind = KIND_MAP[adjusted.sub('swift.', '')] + raise "Unknown symbol kind '#{kind}'" unless sourcekit_kind + self.kind = 'source.lang.swift.decl.' + sourcekit_kind + end + + # Mapping SymbolGraph's ACL to SourceKit + + def init_acl(acl) + self.acl = 'source.lang.swift.accessibility.' + acl + end + + # Symbol location - only available for public+ decls + + def init_location(loc_hash) + self.location = {} + location[:filename] = loc_hash[:uri].sub(%r{^file://}, '') + location[:line] = loc_hash[:position][:line] + location[:character] = loc_hash[:position][:character] + end + + # Generic constraints: in one or both of two places. + # There can be duplicates; these are removed by `Constraint`. + def init_constraints(hash, raw_decl) + raw_constraints = %i[swiftGenerics swiftExtension].flat_map do |key| + next [] unless container = hash[key] + container[:constraints] || [] + end + + constraints = + Constraint.new_list_for_symbol(raw_constraints, path_components) + if raw_decl =~ / where (.*)$/ + constraints += + Constraint.new_list_from_declaration(Regexp.last_match[1]) + end + + self.constraints = constraints.sort.uniq + end + + # Generic type params + def init_generic_type_params(hash) + self.generic_type_params = Set.new( + if (generics = hash[:swiftGenerics]) && + (parameters = generics[:parameters]) + parameters.map { |p| p[:name] } + else + [] + end, + ) + end + + def init_doc_comments(comments_hash) + self.doc_comments = + comments_hash[:lines].map { |l| l[:text] } + .join("\n") + end + + # Availability + # Re-encode this as Swift. Should really teach Jazzy about these, + # could maybe then do something smarter here. + def init_availability(avail_hash_list) + self.availability = avail_hash_list.map do |avail| + str = '@available(' + if avail[:isUnconditionallyDeprecated] + str += '*, deprecated' + elsif domain = avail[:domain] + str += domain + %i[introduced deprecated obsoleted].each do |event| + if version = avail[event] + str += ", #{event}: #{decode_version(version)}" + end + end + else + warn "Found confusing availability: #{avail}" + next nil + end + + str += ', message: "' + avail[:message] + '"' if avail[:message] + str += ', renamed: "' + avail[:renamed] + '"' if avail[:renamed] + + str + ')' + end.compact + end + + def decode_version(hash) + str = hash[:major].to_s + str += ".#{hash[:minor]}" if hash[:minor] + str += ".#{hash[:patch]}" if hash[:patch] + str + end + + # Sort order + include Comparable + + # rubocop:disable Metrics/CyclomaticComplexity + # rubocop:disable Metrics/PerceivedComplexity + def <=>(other) + # Things with location: order by file/line/column + # (pls tell what wheel i am reinventing :/) + if location && other_loc = other.location + if location[:filename] == other_loc[:filename] + if location[:line] == other_loc[:line] + return location[:character] <=> other_loc[:character] + end + return location[:line] <=> other_loc[:line] + end + return location[:filename] <=> other_loc[:filename] + end + + # Things with a location before things without a location + return +1 if location.nil? && other.location + return -1 if location && other.location.nil? + + # Things without a location: by name and then USR + return usr <=> other.usr if name == other.name + name <=> other.name + end + # rubocop:enable Metrics/PerceivedComplexity + # rubocop:enable Metrics/CyclomaticComplexity + end + end +end +# rubocop:enable Metrics/ClassLength diff --git a/spec/integration_spec.rb b/spec/integration_spec.rb index 7bd880b97..963005a46 100644 --- a/spec/integration_spec.rb +++ b/spec/integration_spec.rb @@ -174,7 +174,7 @@ def configure_cocoapods "-I #{base} " \ '-fmodules' `#{sourcekitten} doc --objc #{objc_args} > objc.json` - `#{sourcekitten} doc > swift.json` + `#{sourcekitten} doc -- clean build > swift.json` end behaves_like cli_spec 'misc_jazzy_objc_features', @@ -224,6 +224,17 @@ def configure_cocoapods describe 'Creates docs for Swift project with a variety of contents' do behaves_like cli_spec 'misc_jazzy_features' end + + describe 'Creates docs for Swift project from a .swiftmodule' do + build_path = Dir.getwd + 'tmp/.build' + package_path = + ROOT + 'spec/integration_specs/misc_jazzy_symgraph_features/before' + `swift build --package-path #{package_path} --build-path #{build_path}` + module_path = `swift build --build-path #{build_path} --show-bin-path` + behaves_like cli_spec 'misc_jazzy_symgraph_features', + '--swift-build-tool symbolgraph ' \ + "--build-tool-arguments -I=#{module_path} " + end end if !spec_subset || spec_subset == 'swift' describe 'jazzy cocoapods' do diff --git a/spec/integration_specs b/spec/integration_specs index dcd3e52ee..bb562f641 160000 --- a/spec/integration_specs +++ b/spec/integration_specs @@ -1 +1 @@ -Subproject commit dcd3e52ee87918f68e083498fe9365ffc0546e74 +Subproject commit bb562f641bf38bd4ddc0a364819057600af5de2d