eth_mac_1g_gmii_fifo with Xilinx 1G/2.5G Ethernet PCS/PMA in 1000BASE-X mode

[tristancmorgan]

Has anyone tried to instantiate the eth_mac_1g_gmii_fifo with a Xilinx 1G/2.5G Ethernet PCS/PMA core? I'm struggling with how to clock the setup properly and could use some input or discussion if anyone can help.

My desired result:

• Two 1G GTYs on the same quad (so they need to share clocking and reset logic),
• Which are Instantiated inside two Xilinx 1G/2.5G Ethernet PCS/PMA cores,
• With two eth_mac_1g_gmii_fifo instantiations, each one connected to one of the 1G/2.5G Ethernet PCS/PMA cores

This simple diagram illustrates what I'm trying to achieve:

The 1G examples in this repo seem to exclusively interface with off-chip PHYs, whereas I want to use an on-chip PHY.

My specific question (though I'll gladly accept any critique) is: How should I wire up the clocks on the eth_mac_1g_gmii_fifo, given:

• I don't have any PHYs other than the 1g one and hence no MII clock,
• and there's no TX clk input on my PHY, and hence nowhere to wire gmii_tx_clk to?

My current best guess is this:

        gtx_clk                        => txuserclk2,
        gmii_rx_clk                 => rxuserclk2, 
        gmii_tx_clk                 => open,
        mii_tx_clk                   => txuserclk2,

And here's my entire module, with boring bits removed:

entity eth_1g is
    generic (
        CHANNEL_ONLY                    : boolean
    );
    port (
        clk_i                           : in std_logic; -- Used for reset and registers
        reset_i                         : in std_logic; -- Reset for everything except the GTY control itself. Synchronous with clk_i

        clk_data_i                      : in std_logic; -- Used for datapath

        clk_ctrl_i                      : in std_logic; -- 125MHz free running clk
        reset_ctrl_i                    : in std_logic; -- Reset the GTY state machine. Synchronous with clk_ctrl_i

        -- Used when not CHANNEL_ONLY
        gtrefclk_p_i    : in std_logic;
        gtrefclk_n_i    : in std_logic;
        gtrefclk_o      : out std_logic;
        txuserclk_o     : out std_logic;
        txuserclk2_o    : out std_logic;
        rxuserclk_o     : out std_logic;
        rxuserclk2_o    : out std_logic;
        pma_reset_o     : out std_logic;
        mmcm_locked_o     : out std_logic;
        -----------------------------

        -- Used when CHANNEL_ONLY
        gtrefclk_i      : in std_logic;
        txuserclk_i     : in std_logic;
        txuserclk2_i    : in std_logic;
        rxuserclk_i     : in std_logic;
        rxuserclk2_i    : in std_logic;
        pma_reset_i     : in std_logic;
        mmcm_locked_i     : in std_logic;
        -----------------------------

        tx_p_o                          : out std_logic;
        tx_n_o                          : out std_logic;

        rx_p_i                          : in std_logic;
        rx_n_i                          : in std_logic;

        clk_tx_o                        : out std_logic;
        clk_rx_o                        : out std_logic;

        tx_data_i                       : in axis_64_packed_t;
        tx_ready_o                      : out std_logic;

        rx_data_o                       : out axis_64_packed_t;
        rx_ready_i                      : in std_logic;

        status_o                        : out eth_1g_status_packed_t
    );
end;

architecture arch_imp of eth_1g is
   
    signal reset_in_data_clk_domain         : std_logic;
    signal reset_in_rx_clk_domain           : std_logic;
    signal reset_in_tx_clk_domain           : std_logic;

    signal rx_data                              : axis_64_t;

    signal gmii_txd                         : std_logic_vector(7 downto 0);
    signal gmii_tx_en                       : std_logic;
    signal gmii_tx_er                       : std_logic;
    signal gmii_rxd                         : std_logic_vector(7 downto 0);
    signal gmii_rx_dv                       : std_logic;
    signal gmii_rx_er                       : std_logic;

    signal count_cycles_no_rx_errors    : unsigned(clog2(RX_WAIT_CYCLES_UNTIL_MARK_LINK_STABLE + 1) - 1 downto 0);
    signal rx_error_bad_frame           : std_logic;

    signal txuserclk2                   : std_logic;
    signal rxuserclk2                   : std_logic;
begin
    g_pma_pcs : if CHANNEL_ONLY generate
        i_pcs_pma_channel : eth_1g_pcs_pma_channel
        port map (
            gtrefclk => gtrefclk_i,
            txn => tx_n_o,
            txp => tx_p_o,
            rxn => rx_n_i,
            rxp => rx_p_i,
            independent_clock_bufg => clk_ctrl_i,
            txoutclk => open,
            gtpowergood => open,
            rxoutclk => open,
            resetdone => open,
            cplllock => open,
            mmcm_reset => open,
            userclk => txuserclk_i,
            userclk2 => txuserclk2_i,
            pma_reset => pma_reset_i,
            mmcm_locked => mmcm_locked_i,
            rxuserclk => rxuserclk_i,
            rxuserclk2 => rxuserclk2_i,
            gmii_txd => gmii_txd,
            gmii_tx_en => gmii_tx_en,
            gmii_tx_er => gmii_tx_er,
            gmii_rxd => gmii_rxd,
            gmii_rx_dv => gmii_rx_dv,
            gmii_rx_er => gmii_rx_er,
            gmii_isolate => open,
            configuration_vector => "00001", -- Enable TX even if no RX
            status_vector => open,
            reset => reset_ctrl_i,
            signal_detect => '1'
        );

        -- Kinda weird routing inside the core means that txuserclk_i and rxuserclk2_i have the same freq (62.5MHz)
        clk_tx_o <= txuserclk_i;
        clk_rx_o <= rxuserclk2_i;

        txuserclk2 <= txuserclk2_i;
        rxuserclk2 <= rxuserclk2_i;
    else generate
        i_pcs_pma_full : eth_1g_pcs_pma_full
        port map (
            -- Transceiver ports
            gtrefclk_p => gtrefclk_p_i,
            gtrefclk_n => gtrefclk_n_i,
            gtrefclk_out => gtrefclk_o,
            txn => tx_n_o,
            txp => tx_p_o,
            rxn => rx_n_i,
            rxp => rx_p_i,
            
            -- Recovered clocks
            userclk_out => txuserclk_o,
            userclk2_out => txuserclk2_o,
            rxuserclk_out => rxuserclk_o,
            rxuserclk2_out => rxuserclk2_o,

            independent_clock_bufg => clk_ctrl_i,

            gtpowergood => open,
            resetdone => open,
            pma_reset_out => pma_reset_o,
            mmcm_locked_out => mmcm_locked_o,
            reset => reset_ctrl_i,

            -- GMII interface
            gmii_txd => gmii_txd,
            gmii_tx_en => gmii_tx_en,
            gmii_tx_er => gmii_tx_er,
            gmii_rxd => gmii_rxd,
            gmii_rx_dv => gmii_rx_dv,
            gmii_rx_er => gmii_rx_er,

            -- Control
            gmii_isolate => open,
            configuration_vector => "00001", -- Enable TX even if no RX
            status_vector => open,
            signal_detect => '1'
        );

        -- Kinda weird routing inside the core means that txuserclk_o and rxuserclk2_o have the same freq (62.5MHz)
        clk_tx_o <= txuserclk_o;
        clk_rx_o <= rxuserclk2_o;

        txuserclk2 <= txuserclk2_o;
        rxuserclk2 <= rxuserclk2_o;
    end generate;

    i_reset_in_data_domain : entity verilog_ethernet.sync_reset
    generic map (
        N => 4
    )
    port map (
        clk_i   => clk_data_o,
        reset_i => reset_i,
        reset_o => reset_in_data_clk_domain
    );

    i_mac : entity verilog_ethernet.eth_mac_1g_gmii_fifo
    generic map (
        IODDR_STYLE                 => "IODDR",
        CLOCK_INPUT_STYLE           => "BUFG",
        AXIS_DATA_WIDTH             => 64
    )
    port map (
        gtx_clk                     => txuserclk2, -- The Xilinx docs say this will be 125MHz when the PHY is configured for 1G
        gtx_rst                     => reset_ctrl_i, -- Should probably sync this to gtx_clk but the tools are not complaining...
        logic_clk                   => clk_data_i, -- Whatever I want it to be
        logic_rst                   => reset_in_data_clk_domain,

        gmii_rx_clk                 => rxuserclk2, 
        gmii_tx_clk                 => open,
        mii_tx_clk                  => txuserclk2,

        tx_axis_tdata               => unpack(tx_data_i).data,
        tx_axis_tkeep               => unpack(tx_data_i).keep,
        tx_axis_tvalid              => unpack(tx_data_i).valid,
        tx_axis_tready              => tx_ready_o,
        tx_axis_tlast               => unpack(tx_data_i).last,
        tx_axis_tuser               => ( others => '0' ),

        rx_axis_tdata               => rx_data.data,
        rx_axis_tkeep               => rx_data.keep,
        rx_axis_tvalid              => rx_data.valid,
        rx_axis_tready              => rx_ready_i,
        rx_axis_tlast               => rx_data.last,
        rx_axis_tuser               => open,

        gmii_txd                    => gmii_txd,
        gmii_tx_en                  => gmii_tx_en,
        gmii_tx_er                  => gmii_tx_er,
        gmii_rxd                    => gmii_rxd,
        gmii_rx_dv                  => gmii_rx_dv,
        gmii_rx_er                  => gmii_rx_er,

        tx_error_underflow          => tx_error_underflow,
        rx_error_bad_frame          => rx_error_bad_frame,
        rx_error_bad_fcs            => rx_error_bad_fcs,

        cfg_ifg                     => std_logic_vector(to_unsigned(12, 8)), -- Not sure what the IFG is supposed to be for 1g
        cfg_tx_enable               => '1',
        cfg_rx_enable               => '1'
    );
end;

I have a build with this configuration which I intend to try tomorrow, but I'd also very much appreciate some guidance so I know I'm on the right track (and can exclude the clocking from the inevitable debugging that will follow).

[alexforencich]

You want this module instead: https://github.com/alexforencich/verilog-ethernet/blob/master/rtl/eth_mac_1g_fifo.v

[alexforencich]

I also recommend looking at the VCU108/VCU118 designs, which use the SGMII core. IIRC the RX data is crossed into the TX clock domain inside of the PHY core, so everything runs in txusrclk2.

[tristancmorgan]

Oh nice, thanks for the quick reply. For clarity: is eth_mac_1g_fifo the correct module because the "tri-mode-ness" of the MAC and PHY need to match, and I have a 1G (non-tri-mode) PHY which then should match the 1G (non-tri-mode) MAC provided by the eth_mac_1g_fifo?

Also I think you're right about the RX data being crossed onto the TX clock domain in the PHY core, I found a footnote in the user guide (PG047): "[the rx gmii signals] are synchronous to the internal 125 MHz reference clock of the core. This is userclk2 [(i.e. the tx userclk2)] when the core is used with the device-specific transceiver".

I'll change the module, correct my clocking on the RX GMII signals, check out the example designs you suggest, and see how it goes :)

[alexforencich]

No, the GMII version also contains the IOB logic for GMII, which you need for an external GMII PHY chip. This other module is just the bare MAC without any additional PHY interface logic.

[alexforencich]

Also, the SGMII designs have a clock enable connected to support the lower rates. Just tie that high on the MAC, and tie the MII select signals low.