Troubleshooting solar sense - rev 1

This document outlines all my troubleshooting with solar sense:

• When I first got the board I noticed a couple of issues including a short to 12V to GND and incorrect relay coil logic - that was resolved and documented on the solar sense changes page

• After assembling everything I would need for solar I started trying out SPI for the first time which was obviously a complete failure → pwm was 0x30 which I told wasn't possible and the only line I saw working correctly was CS which was disconnected from the spi bus (all the other lines were permanently low and even clock should have idled high). This pic is the CS line

• After finding all the spi pins weren't working I switched to the SPI smoke test from spv1020 smoke test and was trying to figure out if it was a CB issue. I then scoped the pins on the CB and found that after configuring the smoke test for a looping constant of 2000 (20000 was too high and 2 was too low) I could actually capture the clock on my scope (first image is the 2000 sections of clock period second image is a close up on one of the periods)

• I was also able to see a small MOSI period from the CB

• Next I figured something was wrong solar sense side. I reflowed everything so far and found that the last thing I reflowed was the issue → somejow I had caused a mosi to GND short on the isolator and a MISO to CS short on the CB connectoromce that was solved on the CB side of the isolator I was able to see clock, CS and MOSI correctly! (Image is MOSI)

• SPV1020 smoke test still reads nothing from the SPV1020 (All 0s) and fyi my setup has two battery packs on the input of the Nomura MPPT with around 22V nominal and when connected the mppt boosts the voltage to 25V. No load rn as I'm just trying to get v sense values. When I probe for the digital signals on the other side of the isolator I see nothing 😑 I may have destroyed it while reflowing. Validation to be continued this morning.

• continuing in the morning, i realize i was most likely probing the ground on the CB side instead of the isolated side (it was 12:30am) so actually it does seem to see the Clock on the isolated side 😃 (See the clock in the image below)

• However, the MOSI signal cannot be found on the isolated side so i probed around and found that it is shorted to gnd on the isolated side 😞 But I’ve isolated the short to MOSI and gnd on the mppt

• I found out this ground was my unfortunate issue with cutting the traces to the gnd jumpers on the MPPT to disable the SPI header sigh. It was difficult because those traces are so short that the header soldered on top makes the headers unable to be accessed. Therefore, I had to cut a rectangle around the header instead and it wasn’t fully cut until right now. Now i can see MOSI on the isolated side 😃 I also see some non zero output on SPI on minicom with the same setup as before:

[0] projects/smoke_spv1020/src/main.c:100 SPV1020 #5 voltage_in is: 0x0
[0] projects/smoke_spv1020/src/main.c:107: SPV1020 #5 current is: 0x0
[0] projects/smoke_spv1020/src/main.c:86: SPV1020 #5 status is: 0x2
[0] projects/smoke_spv1020/src/main.c:93: SPV1020 #5 pwm is: 0x3432
[0] projects/smoke_spv1020/src/main.c:100 SPV1020 #5 voltage_in is: 0x0
[0] projects/smoke_spv1020/src/main.c:107: SPV1020 #5 current is: 0x0
[0] projects/smoke_spv1020/src/main.c:86: SPV1020 #5 status is: 0x2
[0] projects/smoke_spv1020/src/main.c:93: SPV1020 #5 pwm is: 0x3432
[0] projects/smoke_spv1020/src/main.c:100 SPV1020 #5 voltage_in is: 0x0
[0] projects/smoke_spv1020/src/main.c:107: SPV1020 #5 current is: 0x0
[0] projects/smoke_spv1020/src/main.c:86: SPV1020 #5 status is: 0x2

• several steps later and other shorts taken out, old MPPT was going into a OTP or triggered the CR bit somehow with 0x02 status, so I switched the mppt to see if it was causing an issue. The output of new mppt after rewiring it seems to actually record accurate voltage, status and PWM readings (no load so current readings are 0). See below for different data points I measured:

This testing was continued sept 8th:

i placed a really big (can dissipate a lot of power) 15 ohm resistance on the output of the mppt across from the input of where the fuse should be and the through hole where the output of the 15th mppt is. I looked at the current readings over spi and found that they were constantly switching between 0 and a number, werent entirely accurate, but did change as I increased and decreased the current across the load (only for a range of about 600mA to 800mA). Note that the power source is a power supply because for some reason my battery harness isnt working rn. Some of the inaccuracy can be attributed to using a power supply source without caps to smoothen the ripple.

heres a sample log from minicom:

[0] projects/smoke_spv1020/src/main.c:107: SPV1020 #5 current is: 0x2e
[0] projects/smoke_spv1020/src/main.c:86: SPV1020 #5 status is: 0x0
[0] projects/smoke_spv1020/src/main.c:93: SPV1020 #5 pwm is: 0x56
[0] projects/smoke_spv1020/src/main.c:100: SPV1020 #5 voltage_in is: 0x2c9
[0] projects/smoke_spv1020/src/main.c:107: SPV1020 #5 current is: 0x2e
[0] projects/smoke_spv1020/src/main.c:86: SPV1020 #5 status is: 0x0
[0] projects/smoke_spv1020/src/main.c:93: SPV1020 #5 pwm is: 0x66
[0] projects/smoke_spv1020/src/main.c:100: SPV1020 #5 voltage_in is: 0x2ca
[0] projects/smoke_spv1020/src/main.c:107: SPV1020 #5 current is: 0x0
[0] projects/smoke_spv1020/src/main.c:86: SPV1020 #5 status is: 0x0
[0] projects/smoke_spv1020/src/main.c:93: SPV1020 #5 pwm is: 0x56
[0] projects/smoke_spv1020/src/main.c:100: SPV1020 #5 voltage_in is: 0x2ca
[0] projects/smoke_spv1020/src/main.c:107: SPV1020 #5 current is: 0x2e
[0] projects/smoke_spv1020/src/main.c:86: SPV1020 #5 status is: 0x0
[0] projects/smoke_spv1020/src/main.c:93: SPV1020 #5 pwm is: 0x56
[0] projects/smoke_spv1020/src/main.c:100: SPV1020 #5 voltage_in is: 0x2c9
[0] projects/smoke_spv1020/src/main.c:107: SPV1020 #5 current is: 0x2e
[0] projects/smoke_spv1020/src/main.c:86: SPV1020 #5 status is: 0x0
[0] projects/smoke_spv1020/src/main.c:93: SPV1020 #5 pwm is: 0x66
[0] projects/smoke_spv1020/src/main.c:100: SPV1020 #5 voltage_in is: 0x2ca
[0] projects/smoke_spv1020/src/main.c:107: SPV1020 #5 current is: 0x0

However since we have tested these mppts out before and have gotten both accurate voltage and current readings, this investigation for hardware functionality with the SPV1020 can be closed out (FW still needs to process the reads off the registers I think).

• Moving on for now to MCP3427 validation, I cannot probe any I2C on any of the pins and see traffic. I then realized that i put the pullup resistors on the other side of the board which was not a good idea (R33 and R32) so i made a note of that for revisions, and will populate and test.

• well, still no communication. I do see that both sda and scl are just staying at 3v3 (thanks to the pullups) but I dont see them being driven low by the I2C pins.

• i touched up the controller board pins with solder and finally see something on SDA! (nothing on scl yet). An important note is to not hand assemble the next rev of solar sense and use a stencil instead.

• After many hours later, i finally fixed the scl contact issue with the controller board connector 😑 i will 100% use a stencil next time.

• I can now see communication with clock and data on CB side. (see images - SDA & SCL)

• i also see a clock and data on the isolated side, but it is not clean. The following image is the clock and the isolated 3v3 rail has a large ripple (maybe add a larger capacitance to that rail) and there's a long rise time to clock. If this seems to be an issue, I will measure the time to rise and compare with the datasheet.

and data:

• i noticed that even though SCL and SDA reached the mcp3427, it still could not communicate. I then looked through the datasheet and saw this table

and noticed our clock was high about 0.5us which does not meet the min timing requirements

• turns out we use a fast I2C mode at 400kHz so the above two images and statements dont matter

• the mcp2347 is still giving a timeout as follows:

[0] libraries/ms-common/src/stm32f0xx/i2c.c:86: Timeout: 2 waiting for 0 to change
[0] libraries/ms-common/src/stm32f0xx/i2c.c:86: Timeout: 2 waiting for 0 to change
[0] libraries/ms-common/src/stm32f0xx/i2c.c:86: Timeout: 2 waiting for 0 to change
[0] libraries/ms-common/src/stm32f0xx/i2c.c:86: Timeout: 2 waiting for 0 to change

In Timeout: 2 waiting for 0 to change , it’s timing out while waiting for the TXIS flag (id 2) on the stm32 to change from 0. TXIS is set to 1 when it receives the ack pulse on SDA from the i2c slave after each data byte is tx’d. So we think the mcp3427 is never sending the ack pulse

• Still cant get I2C to work, so I started probing to see if maybe theres a signal integrity issue. The green is SCL and yellow is SDA. If this is a signal integrity issue, it will be investigated tomorrow (or I guess today since its 2am)

• to continue the troubleshooting process and see if the signal integrity was the issue, I connected the current sense mcp3247 that does not have to go through any layers of isolation and gets its analog input from the hall effect sensor. for this testing, the actual voltage on the input didnt matter too much as the hall effect wasnt being tested, but the adc was. I figured out there was a CB solder problem (again) and then was able to see SDA and SCL on the lines to the U3 mcp.

SCL:

SDA:

Although the lines are apparent, there was still no communication from minicom 😞 . This leads me to believe that it might be a FW issue rather than signal integrity, as both these MCPs failed to communicate. Investigation to be continued.

Another day in the solar sense troubleshooting, and I just figured out how to decode I2C on the scope I was using. I set everything up only to see the 3V3 line go high then slowly decrease to GND potential. After a while (~1hr) it was clearly shorted to GND. I then spent the next ~2hrs trying to find the short, and then eventually removing every component on the 3V3 rail with subsequent checks if that fixed the short. It did not.

I then removed the solder from every pad on any component interfacing with 3V3. There was still a short until I removed the solder from the pads of U2 and C6. I then did not see a short (and ofc this was these were the last pads which solder I removed). Turns out there was a short to GND on pad 1 of C6 to the GND plane. This was due to the clearance from polypours which as can see from the image below, is not very large. That coupled with manufacturing causes a short with solder on the pad, and so i cut the right side of that pad. After this point, it is fully isolated. This was a very unfortunate issue (as I still have to put all the components back on including CB connector) but at least I now know for rev 2 that I should put more clearance.

Finally, the day has come when i figure out the communication issue. the next points outline what happened so i know for next time and hopefully dont have to ever do this again. TL;DR: i did bad soldering on the MCP3427 (no shorts but not all pins were soldered down - i attribute this to the difficulties of quarantine soldering/assembly)

• first, I fixed the short issue above by taking the scalpel to the board several more times until it was impossible to short.

• I then still could not see the communication, but could see the 3v3 rail on those pins which was a good sign.

• I realized after more debugging (longer than it should have taken) that I had commented out the i2c port 2 init command in the firmware smoke test, so that was a big oof

• next, I saw the following from debugging I2C on my scope.

• from the above images, it can be seen that the correct bits were sent over SDA (6B is correct), but the red bit at the end signafies there was no acknowledge

• this was also probed on the isolated side, so there was definitely not enough SI issues because it could still be decoded

• therefore, the issue was with the MCP3427

• after probing for more time, I realized the issue with the MCP3427 was that some of the pins werent actually soldered completely down to the pads. Im going to blame this on the fact i had to hand solder everything in an unideal station, but its still a good note for the future.

• In any case, I flashed the code one more time, and finally there was communication!!!

[0] projects/smoke_mcp3427/src/main.c:150: Voltage sense mcp3427 ID = 5; Value = -16
[0] projects/smoke_mcp3427/src/main.c:150: Voltage sense mcp3427 ID = 5; Value = -16
[0] projects/smoke_mcp3427/src/main.c:150: Voltage sense mcp3427 ID = 5; Value = -16
[0] projects/smoke_mcp3427/src/main.c:150: Voltage sense mcp3427 ID = 5; Value = -16
[0] projects/smoke_mcp3427/src/main.c:150: Voltage sense mcp3427 ID = 5; Value = -16
[0] projects/smoke_mcp3427/src/main.c:150: Voltage sense mcp3427 ID = 5; Value = -16
[0] projects/smoke_mcp3427/src/main.c:150: Voltage sense mcp3427 ID = 5; Value = -16
[0] projects/smoke_mcp3427/src/main.c:150: Voltage sense mcp3427 ID = 5; Value = -16
[0] projects/smoke_mcp3427/src/main.c:150: Voltage sense mcp3427 ID = 5; Value = -16
[0] projects/smoke_mcp3427/src/main.c:150: Voltage sense mcp3427 ID = 5; Value = -16
[0] projects/smoke_mcp3427/src/main.c:150: Voltage sense mcp3427 ID = 5; Value = -16
[0] projects/smoke_mcp3427/src/main.c:150: Voltage sense mcp3427 ID = 5; Value = -16
[0] projects/smoke_mcp3427/src/main.c:150: Voltage sense mcp3427 ID = 5; Value = -16
[0] projects/smoke_mcp3427/src/main.c:150: Voltage sense mcp3427 ID = 5; Value = -16
[0] projects/smoke_mcp3427/src/main.c:150: Voltage sense mcp3427 ID = 5; Value = -16
[0] projects/smoke_mcp3427/src/main.c:150: Voltage sense mcp3427 ID = 5; Value = -16
[0] projects/smoke_mcp3427/src/main.c:150: Voltage sense mcp3427 ID = 5; Value = -16

• This was the output with no mppt and 0V on the vsense pin, so the next steps are to see how accurate it is with different nomura input voltages. In any case, seeing i2c reads is such a relief.