Refinancing and loan modifications can affect your FICO score in a few areas. How much depends on whether it's reported to the credit bureaus as the same loan with changes or as an entirely new loan.
If it's reported as the same loan with changes, three pieces of information associated with the loan modification may affect your score: the credit inquiry, changes to the loan balance, and changes to the terms of that loan. Overall, the impact of these changes on your FICO score should be minimal.
If it's reported as a "new" loan, your score could still be affected by the inquiry, balance, and terms of the loan, along with the additional impact of a new "open date." A new or recent open date typically indicates that it is a new credit obligation and, as a result, can impact the score more than if the terms of the existing loan are simply changed.